Augmented multi-tier classifier for multi-modal voice activity detection

1. A method comprising:

at a first time:

receiving, from a first classifier, a first modality output detected from a voice input from a human subject; and

receiving, from a second classifier, a second modality output detected from visual input from the human subject,

wherein the voice input and the visual input are provided by the human subject at a same time, and wherein a mix and a weighting for the first classifier and the second classifier is based on speaking patterns for a given time of day for the human subject;

providing, via a feedback loop, the first modality output to the second classifier, to yield an updated second classifier, wherein the first classifier and the updated second classifier comprise a classifier group;

at a second time, after the first time:

receiving classifier group output from the classifier group;

concatenating, via a third classifier, the classifier group with original features of the human subject, to yield a classifier output, wherein the original features of the human subject comprise a temporal alignment of voice data associated with the first modality output and visual data associated with the second modality output; and

determining voice activity based on the classifier output.

2. The method of claim 1 , wherein an acoustic feature associated with the voice input comprises one of Mel-frequency cepstral coefficients, a first derivative of Mel-frequency cepstral coefficients, a second derivative of Mel-frequency cepstral coefficients, and acoustic energy.

3. The method of claim 1 , wherein a visual feature associated with the visual input comprises one of a parametric feature, an appearance-based feature, dimensions of a mouth region, image region intensity, discrete cosine transformations, image saturation, image brightness, image texture, video motion, head movement, one of movement of a specific item relative to other items and movement of the specific item relative to a background, micro-expressions on a person's face, extremity of expression, detected lighting changes, head angle relative to a camera, and local binary patterns.

4. The method of claim 1 , wherein the original features are associated with a video frame.

5. The method of claim 1 , wherein concatenating the classifier group output with original features of the human subject further comprises:

performing late fusion of the original features using weighted majority voting.

6. A system comprising:

a processor; and

a computer-readable medium having instructions which, when executed by the processor, cause the processor to perform operations comprising:

at a first time:

receiving, from a first classifier, a first modality output detected from a voice input from a human subject; and

receiving, from a second classifier, a second modality output detected from visual input from the human subject,

wherein the voice input and the visual input are provided by the human subject at a same time, and wherein a mix and a weighting for the first classifier and the second classifier is based on speaking patterns for a given time of day for the human subject;

providing, via a feedback loop, the first modality output to the second classifier, to yield an updated second classifier, wherein the first classifier and the updated second classifier comprise a classifier group;

at a second time, after the first time:

receiving classifier group output from the classifier group;

concatenating, via a third classifier, the classifier group output with original features of the human subject, to yield a classifier output, wherein the original features of the human subject comprise a temporal alignment of voice data associated with the first modality output and visual data associated with the second modality output; and

determining voice activity based on the classifier output.

7. The system of claim 6 , wherein an acoustic feature associated with the voice input comprises one of Mel-frequency cepstral coefficients, a first derivative of Mel-frequency cepstral coefficients, a second derivative of Mel-frequency cepstral coefficients, and acoustic energy.

8. The system of claim 6 , wherein a visual feature associated with the visual input comprises one of a parametric feature, an appearance-based feature, dimensions of a mouth region, image region intensity, discrete cosine transformations, image saturation, image brightness, image texture, video motion, head movement, one of movement of a specific item relative to other items and movement of the specific item to a background, micro-expressions on a person's face, extremity of expression, detected lighting changes, head angle relative to a camera, and local binary patterns.

9. The system of claim 6 , wherein the original features are associated with a video frame.

10. The system of claim 6 , the computer-readable medium having additional instructions stored which, when executed by the processor, causes the processor to perform operation comprising:

performing, during the concatenating, late fusion of the original features using weighted majority voting.

11. A computer-readable storage medium storing instructions which, when executed by a computing device, cause the computing device to perform operations comprising:

at a first time:

receiving, from a first classifier, a first modality output detected from a voice input from a human subject; and

receiving, from a second classifier, a second modality output detected from visual input from the human subject,

wherein the voice input and the visual input are provided by the human subject at a same time, and wherein a mix and a weighting for the first classifier and the second classifier is based on speaking patterns for a given time of day for the human subject;

providing, via a feedback loop, the first modality output to the second classifier, to yield an updated second classifier, wherein the first classifier and the updated second classifier comprise a classifier group;

at a second time, after the first time:

receiving classifier group output from the classifier group;

concatenating, via a third classifier, the classifier group output with original features of the human subject, to yield a classifier output, wherein the original features of the human subject comprise a temporal alignment of voice data associated with the first modality output and visual data associated with the second modality output; and

determining voice activity based on the classifier output.

12. The computer-readable storage medium of claim 11 , wherein an acoustic feature associated with the voice input comprises one of Mel-frequency cepstral coefficients, a first derivative of Mel-frequency cepstral coefficients, a second derivative of Mel-frequency cepstral coefficients, and acoustic energy.

13. The computer-readable storage medium of claim 11 , wherein a visual feature associated with the visual input comprises one of a parametric feature, an appearance-based feature, dimensions of a mouth region, image region intensity, discrete cosine transformations, image saturation, image brightness, image texture, video motion, head movement, one of movement of a specific item relative to other items and movement of the specific item to a background, micro-expressions on a person's face, extremity of expression, detected lighting changes, head angle relative to a camera, and local binary patterns.

14. The computer-readable storage medium of claim 11 , wherein the original features are associated with a video frame.