System and method for audio-visual speech recognition
View Patent ↗Disclosed herein is method of performing speech recognition using audio and visual information, where the visual information provides data related to a person's face. Image preprocessing identifies regions of interest, which is then combined with the audio data before being processed by a speech recognition engine.
1. A method of performing speech recognition at a distance comprising:
obtaining audio information from a plurality of microphones positioned at a distance from a speaker;
obtaining visual information from an image capture device;
pre-processing the visual information, wherein pre-processing comprises:
using a recurrent deep neural network model to identify relevant image features by classifying a pixel in a first frame of the visual information using features from a pixel in the same location from a prior frame of the visual information;
identifying a region of interest in the visual information; and
using the region of interest, aligning the visual information with the acoustic information and classifying individual frames in the visual information into context-dependent phonetic states;
combining the audio information and visual information within a single deep neural network classifier; and
performing a speech recognition process on the combined audio information and visual information, wherein the speech recognition process comprises:
generating observation probabilities for context-dependent phonetic states using a joint audio-visual observation model, and
conducting a search in a standard speech recognition engine using the observation probabilities.
2. The method of claim 1 , wherein pre-processing further comprises:
determining whether a speaker is present in the visual information.
3. The method of claim 1 , wherein the standard speech recognition engine is a WFST-based speech recognition engine.
4. The method of claim 1 , wherein pre-processing further comprises:
defining the region of interest by generating a probability distribution of classes for each pixel in the visual information.
5. The method of claim 1 , further comprising:
identifying a location of the speaker relative to the image capture device.
6. The method of claim 1 , wherein pre-processing further comprises:
scaling the region of interest.
7. The method of claim 1 , wherein combining the audio information and visual information comprises:
combining the audio information and the visual information in an output layer of the single deep neural network classifier.
8. The method of claim 1 , wherein combining the audio information and visual information comprises:
combining the audio information and the visual information in a second hidden layer of the single deep neural network classifier.