System and method for enhancing speech activity detection using facial feature detection
View Patent ↗Disclosed herein are systems, methods, and non-transitory computer-readable storage media for processing audio. A system configured to practice the method monitors, via a processor of a computing device, an image feed of a user interacting with the computing device and identifies an audio start event in the image feed based on face detection of the user looking at the computing device or a specific region of the computing device. The image feed can be a video stream. The audio start event can be based on a head size, orientation or distance from the computing device, eye position or direction, device orientation, mouth movement, and/or other user features. Then the system initiates processing of a received audio signal based on the audio start event. The system can also identify an audio end event in the image feed and end processing of the received audio signal based on the end event.
1. A method, comprising:
capturing video content via a video device;
detecting whether a user is depicted in the video content and detecting that a device is also depicted in the video content; and
when the user is present in the video content and the device also present in the video content, transmitting a signal to the device to initiate an audio start event associated with receiving a voice signal from the user.
2. The method of claim 1 , wherein the device is a mobile device.
3. The method of claim 1 , wherein the video content shows the user looking at a specific region of the device.
4. The method of claim 1 , wherein the audio start event is initiated based on one or more of a head orientation of the user, an eye position of the user, an eye direction of the user, a device orientation of the device.
5. The method of claim 1 , further comprising:
determining an audio end event based on the video content; and
causing the device to cease the processing of the voice signal based on the audio end event.
6. The method of claim 5 , wherein the audio end event is initiated based on a determination that the video content shows the user looking away from the device.
7. The method of claim 1 , wherein the device performs speech recognition on the voice signal.
8. The method of claim 1 , wherein the device transmits the voice signal to a second device to process the voice signal.
9. The method of claim 1 , further comprising ignoring a portion of an audio signal that is received prior to the audio start event, the audio signal including the voice signal.
10. An apparatus comprising:
a processor; and
a non-transitory computer-readable storage medium having instructions stored which, when executed by the processor, cause the processor to perform operations comprising:
capturing video content via a video device;
detecting whether a user is depicted in the video content and detecting that a device is also depicted in the video content; and
when the user is present in the video content and the user is looking at the device also present in the video content, transmitting a signal to the computing device to initiate an audio start event associated with receiving a voice signal from the user.
11. The method of claim 10 , wherein the device is a mobile device.
12. The apparatus of claim 10 , wherein the video content shows the user looking at a specific region of the device.
13. The apparatus of claim 10 , wherein the audio start event is initiated based on one or more of a head orientation of the user, an eye position of the user, an eye direction of the user, a device orientation of the device.
14. The apparatus of claim 10 , wherein the non-transitory computer-readable storage medium stores additional instructions which, when executed by the processor, cause the processor to perform operations further comprising:
determining an audio end event based on the video content; and
causing the device to cease the processing of the voice signal based on the audio end event.
15. The apparatus of claim 14 , wherein the audio end event is initiated based on a determination that the video content shows the user looking away from the device.
16. The apparatus of claim 10 , wherein the device performs speech recognition on the voice signal.
17. The apparatus of claim 10 , wherein the device transmits the voice signal to a second device to process the voice signal.
18. The apparatus of claim 10 , wherein the non-transitory computer-readable storage medium stores additional instructions which, when executed by the processor, cause the processor to perform operations further comprising:
ignoring a portion of an audio signal that is received prior to the audio start event, the audio signal including the voice signal.
19. A non-transitory computer-readable storage device having instructions which, when executed by a processor, cause the processor to perform operations comprising:
capturing video content via a video device;
detecting whether a user is depicted in the video content and detecting that a device is also depicted in the video content; and
when the user is present in the video content and the user is looking at the device also present in the video content, transmitting a signal to the device to initiate an audio start event associated with receiving a voice signal from the user.
20. The non-transitory computer-readable storage device of claim 19 , wherein the video content shows the user looking at a specific region of the device.