PROJECT TITLE :

Sequential Audio-Visual Correspondence With Alternating Diffusion Kernels - 2018

ABSTRACT:

A elementary problem in multimodal signal processing is to quantify relations between 2 different signals with respect to a certain phenomenon. During this Project, we address this downside from a kernel-based mostly perspective and propose a live that's primarily based on affinity kernels constructed separately in every modality. This live is motivated from each a kernel density estimation purpose of read of predicting the signal in one modality based mostly on the other, plus from a statistical model, which implies that prime values of the proposed live are expected when signals highly correspond to each other. Considering an online setting, we tend to propose an economical algorithm for the sequential update of the proposed live, and demonstrate its application to eye-fixation prediction in audio-visual recordings. The goal is to predict locations within a video recording at that people gaze when watching the video. As studies in psychology imply, people tend to looked at the placement of the audio supply, thus that their prediction becomes reminiscent of locating the audio supply within the video. So, we have a tendency to propose to predict eye-fixations as regions among the video with the best correspondence to the audio signal, thereby demonstrating the improved performance of the proposed methodology.