Attention mechanism for coping with acoustic-lips timing mismatch in audiovisual processing

1. A computer-implemented method for determining a context-dependent time shift of audio and video features in an audiovisual stream or file, the method comprising:

receiving audio information and video information of the audiovisual stream or file;

processing the audio information and video information separately to generate a new representation of the audio information, including information relating to features of the audio information, and a new representation of the video information, including information relating to features of the video information; and

mapping features of the audio information and features of the video information using an attention mechanism modeling the context-dependent time shift to identify pairs of audio and video features, wherein the pairs of audio and video features are identified as being synchronized (true) features that contain a recording of a speaker in which the audio information of the recording and the video information of lips of the speaker are synchronized or unsynchronized (false) features otherwise, and wherein the attention mechanism uses a weighted sum of a plurality of consecutive audio frames and a weighted sum of a plurality of consecutive video frames and weights of the attention mechanism are based on a content and context of the audio information and on a content and context of the video information.

2. The method of claim 1 , further comprising:

generating an audio stack from the audio information; and

generating a video stack from the video information.

3. The method of claim 2 , wherein the audio stack comprises mel-frequency cepstral coefficients generated from the audio information and the video stack comprises a plurality of frames of video information.

4. The method of claim 3 , wherein the processing comprises:

processing the audio information using a machine learning method modeling the context-dependent time shift; and

processing the video information using a machine learning method modeling the context-dependent time shift.

5. The method of claim 4 , wherein:

the audio machine learning method uses a gated recurrent units network that uses a plurality of consecutive outputs of the audio gated recurrent units network combined using a weighted function summed to one such that a different weight is given to each one of the consecutive audio frames; and

the video machine learning method uses a gated recurrent units network that uses a plurality of consecutive outputs of the video gated recurrent units network combined using a weighted function summed to one such that a different weight is given to each one of the consecutive video frames.

6. The method of claim 5 , further comprising generating synthetic video information that is synchronized to the received audio information based on temporal features detected in the audio by generating matching visual features that provide temporal correspondence as synchronized features.

7. A system for determining a context-dependent time shift of audio and video features in an audiovisual stream or file, the system comprising a processor, memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor to perform:

receiving audio information and video information of the audiovisual stream or file;

processing the audio information and video information separately to generate a new representation of the audio information, including information relating to features of the audio information, and a new representation of the video information, including information relating to features of the video information; and

mapping features of the audio information and features of the video information using an attention mechanism modeling the context-dependent time shift to identify pairs of audio and video features, wherein the pairs of audio and video features are identified as being synchronized (true) features that contain a recording of a speaker in which the audio information of the recording and the video information of lips of the speaker are synchronized or unsynchronized (false) features otherwise, and wherein the attention mechanism uses a weighted sum of a plurality of consecutive audio frames and a weighted sum of a plurality of consecutive video frames and weights of the attention mechanism are based on a content and context of the audio information and on a content and context of the video information.

8. The system of claim 7 , further comprising:

generating an audio stack from the audio information; and

generating a video stack from the video information.

9. The system of claim 8 , wherein the audio stack comprises mel-frequency cepstral coefficients generated from the audio information and the video stack comprises a plurality of frames of video information.

10. The system of claim 9 , wherein the processing comprises:

processing the audio information using a machine learning method modeling the context-dependent time shift; and

processing the video information using a machine learning method modeling the context-dependent time shift.

11. The system of claim 10 , wherein:

the audio machine learning method uses a gated recurrent units network that uses a plurality of consecutive outputs of the audio gated recurrent units network combined using a weighted function summed to one such that a different weight is given to each one of the consecutive audio frames; and

the video machine learning method uses a gated recurrent units network that uses a plurality of consecutive outputs of the video gated recurrent units network combined using a weighted function summed to one such that a different weight is given to each one of the consecutive video frames.

12. The system of claim 11 , further comprising generating synthetic video information that is synchronized to the received audio information based on temporal features detected in the audio by generating matching visual features that provide temporal correspondence as synchronized features.

13. A computer program product for determining a context-dependent time shift of audio and video features in an audiovisual stream or file, the computer program product comprising a non-transitory computer readable storage having program instructions embodied therewith, the program instructions executable by a computer, to cause the computer to perform a method comprising:

receiving audio information and video information of the audiovisual stream or file;

processing the audio information and video information separately to generate a new representation of the audio information, including information relating to features of the audio information, and a new representation of the video information, including information relating to features of the video information; and

mapping features of the audio information and features of the video information using an attention mechanism modeling a context-dependent time shift to identify pairs of audio and video features, wherein the pairs of audio and video features are identified as being synchronized (true) features that contain a recording of a speaker in which the audio information of the recording and the video information of lips of the speaker are synchronized or unsynchronized (false) features otherwise, and wherein the attention mechanism uses a weighted sum of a plurality of consecutive audio frames and a weighted sum of a plurality of consecutive video frames and weights of the attention mechanism are based on a content and context of the audio information and on a content and context of the video information.

14. The computer program product of claim 13 , further comprising:

generating an audio stack from the audio information; and

generating a video stack from the video information.

15. The computer program product of claim 14 , wherein the audio stack comprises mel-frequency cepstral coefficients generated from the audio information and the video stack comprises a plurality of frames of video information.

16. The computer program product of claim 15 , wherein the processing comprises:

processing the audio information using a machine learning method modeling the context-dependent time shift; and

processing the video information using a machine learning method modeling the context-dependent time shift;

wherein the audio machine learning method uses a gated recurrent units network that uses a plurality of consecutive outputs of the audio gated recurrent units network combined using a weighted function summed to one such that a different weight is given to each one of the consecutive audio frames; and

the video machine learning method uses a gated recurrent units network that uses a plurality of consecutive outputs of the video gated recurrent units network combined using a weighted function summed to one such that a different weight is given to each one of the consecutive video frames.

17. The computer program product of claim 16 , further comprising generating synthetic video information that is synchronized to the received audio information based on temporal features detected in the audio by generating matching visual features that provide temporal correspondence as synchronized features.