Audio-visual speech recognition with scattering operators
View Patent ↗Aspects described herein are directed towards methods, computing devices, systems, and computer-readable media that apply scattering operations to extracted visual features of audiovisual input to generate predictions regarding the speech status of a subject. Visual scattering coefficients generated according to one or more aspects described herein may be used as input to a neural network operative to generate the predictions regarding the speech status of the subject. Predictions generated based on the visual features may be combined with predictions based on audio input associated with the visual features. In some embodiments, the extracted visual features may be combined with the audio input to generate a combined feature vector for use in generating predictions.
1. A method comprising:
receiving, by a computing device, audiovisual input comprising audio input and video input associated with a subject;
extracting, by the computing device, visual features from the video input;
applying, by the computing device, a scattering operation to the extracted visual features to generate a vector of scattering coefficients in a first dimensional space;
providing the vector of scattering coefficients as input to a first neural network for visual processing;
generating, by the computing device and using the first neural network, a first prediction regarding a speech status of the subject based on the vector of scattering coefficients;
providing the audio input to a second neural network for audio processing;
generating, by the computing device and using the second neural network, a second prediction regarding the speech status of the subject based on the audio input;
comparing, by the computing device, a first output of the first neural network with a second output of the second neural network to determine a synchrony state of the audio input and the video input, wherein the synchrony state indicates whether the audio input is in-sync with the video input; and
storing the synchrony state of the audio input and the video input.
2. The method of claim 1 , wherein providing the vector of visual scattering features as input to the first neural network for visual processing comprises:
normalizing the vector of scattering coefficients to generate a first normalized vector of scattering coefficients;
aggregating a plurality of normalized vectors of scattering coefficients including the first normalized vector of scattering coefficients to generate a set of aggregated visual feature vectors; and
providing the set of aggregated visual feature vectors to the first neural network for visual input processing.
3. The method of claim 1 , further comprising:
combining, by the computing device, the first output of the first neural network with the second output of the second neural network to generate a fused audiovisual feature vector based on the audiovisual input;
providing the fused audiovisual feature vector to a third neural network for audiovisual processing; and
generating, by the computing device and using the third neural network, a third prediction regarding the speech status of the subject based on the fused audiovisual feature vector.
4. The method of claim 1 , wherein the video input is sampled at a first frequency and the audio input is sampled at a different second frequency.
5. The method of claim 1 , wherein applying the scattering operation to the extracted visual features to generate a vector of visual scattering features in a first dimensional space comprises:
applying the scattering operation to the extracted visual features to generate a second scattering vector in a second dimensional space; and
projecting the second scattering vector into the first dimensional space to generate the vector of visual scattering features in the first dimensional space,
wherein the second dimensional space is of a higher dimensionality than the first dimensional space.
6. The method of claim 1 , wherein applying the scattering operation to the extracted visual features to generate a vector of visual scattering features in a first dimensional space comprises generating first order scattering coefficients and second order scattering coefficients.
7. The method of claim 1 , wherein the first prediction regarding the speech status of the subject is used to recognize speech content of the audiovisual input.
8. One or more non-transitory computer readable media comprising instructions that, when executed by a processor, cause the processor to perform steps comprising:
receiving, by a computing device, audiovisual input comprising audio input and video input associated with a subject;
extracting, by the computing device, visual features from the video input;
applying, by the computing device, a scattering operation to the extracted visual features to generate a vector of scattering coefficients in a first dimensional space;
providing the vector of scattering coefficients as input to a first neural network for visual processing;
providing the audio input to a second neural network for audio processing;
comparing, by the computing device, a first output of the first neural network with a second output of the second neural network to determine a synchrony state of the audio input and the video input, wherein the synchrony state indicates whether the audio input is in-sync with the video input; and
storing the synchrony state of the audio input and the video input.
9. The computer readable media of claim 8 , wherein providing the vector of visual scattering features as input to the first neural network for visual processing comprises:
normalizing the vector of scattering coefficients to generate a first normalized vector of scattering coefficients;
aggregating a plurality of normalized vectors of scattering coefficients including the first normalized vector of scattering coefficients to generate a set of aggregated visual feature vectors; and
providing the set of aggregated visual feature vectors to the first neural network for visual input processing.
10. The computer readable media of claim 8 , wherein the instructions, when executed by the processor, further cause the processor to perform steps comprising:
combining, by the computing device, the first output of the first neural network with the second output of the second neural network to generate a fused audiovisual feature vector based on the audiovisual input;
providing the fused audiovisual feature vector to a third neural network for audiovisual processing; and
generating, by the computing device and using the third neural network, a third prediction regarding the speech status of the subject based on the fused audiovisual feature vector.
11. The computer readable media of claim 8 , wherein the video input is sampled at a first frequency and the audio input is sampled at a different second frequency.
12. The computer readable media of claim 8 , wherein applying the scattering operation to the extracted visual features to generate a vector of visual scattering features in a first dimensional space comprises:
applying the scattering operation to the extracted visual features to generate a second scattering vector in a second dimensional space; and
projecting the second scattering vector into the first dimensional space to generate the vector of visual scattering features in the first dimensional space,
wherein the second dimensional space is of a higher dimensionality than the first dimensional space.
13. The computer readable media of claim 8 , wherein applying the scattering operation to the extracted visual features to generate a vector of visual scattering features in a first dimensional space comprises generating first order scattering coefficients and second order scattering coefficients.
14. The computer readable media of claim 8 , wherein the instructions, when executed by the processor, further cause the processor to perform steps comprising:
generating, by the computing device and using the first neural network, a first prediction regarding a speech status of the subject based on the vector of scattering coefficients,
wherein the first prediction regarding the speech status of the subject is used to recognize speech content of the audiovisual input.
15. A system comprising:
a processor; and
memory storing instructions that, when executed by the processor, cause the processor to:
receive audiovisual input comprising audio input and video input associated with a subject;
extract visual features from the video input;
apply a scattering operation to the extracted visual features to generate a vector of scattering coefficients in a first dimensional space;
provide the vector of scattering coefficients as input to a first neural network for visual processing;
generate, using the first neural network, a first prediction regarding a speech status of the subject based on the vector of scattering coefficients;
normalize the audio input to generate normalized audio input;
combine the vector of scattering coefficients with the normalized audio input to generate a fused audiovisual feature vector;
provide the fused audiovisual feature vector to a second neural network for audiovisual processing; and
generate, using the second neural network, a second prediction regarding the speech status of the subject based on a second output of the second neural network.
16. The system of claim 15 , wherein the instructions, when executed by the processor, cause the processor to provide the vector of visual scattering features as input to the first neural network for visual processing by causing the processor to:
normalize the vector of scattering coefficients to generate a first normalized vector of scattering coefficients;
aggregate a plurality of normalized vectors of scattering coefficients including the first normalized vector of scattering coefficients to generate a set of aggregated visual feature vectors; and
provide the set of aggregated visual feature vectors to the first neural network for visual input processing.
17. The system of claim 15 , wherein the instructions, when executed by the processor, further cause the processor to:
combine a first output of the first neural network with a second output of the second neural network to generate a fused audiovisual feature vector based on the audiovisual input;
providing the fused audiovisual feature vector to a third neural network for audiovisual processing; and
generating, by the computing device and using the third neural network, a third prediction regarding the speech status of the subject based on the fused audiovisual feature vector.
18. The system of claim 15 , wherein the video input is sampled at a first frequency and the audio input is sampled at a different second frequency.
19. The system of claim 15 , wherein the instructions, when executed by the processor, cause the processor to apply the scattering operation to the extracted visual features to generate a vector of visual scattering features in a first dimensional space by causing the processor to:
apply the scattering operation to the extracted visual features to generate a second scattering vector in a second dimensional space; and
project the second scattering vector into the first dimensional space to generate the vector of visual scattering features in the first dimensional space,
wherein the second dimensional space is of a higher dimensionality than the first dimensional space.
20. The system of claim 15 , wherein the first prediction regarding the speech status of the subject is used to recognize speech content of the audiovisual input.