Audio source separation for multi-channel beamforming based on face detection
This disclosure provides methods, devices, and systems for speech enhancement. The present implementations more specifically relate to utilizing multiple modalities to suppress audio originating from a distractor audio source without distorting audio originating from a target audio source. In some aspects, a speech enhancement system may receive a multi-channel audio signal via a microphone array and may further receive an image associated with a respective frame of the audio signal. The speech enhancement system detects one or more target faces in the image and determines whether the audio frame originates from a target audio source. For example, the speech enhancement system may compare a respective direction of each target face with a direction-of-arrival (DOA) of the audio frame. The speech enhancement system may selectively steer a beam associated with a multi-channel beamformer toward the DOA of the audio frame based on whether the audio frame originates from a target face.
1 . A method of processing audio signals, comprising:
receiving an audio signal via a plurality of microphones;
receiving an image associated with a frame of the audio signal;
detecting one or more faces in the received image;
selecting a number (N) of target faces among the one or more faces detected in the received image;
determining a respective direction of each of the N target faces relative to the plurality of microphones; and
selectively steering a beam associated with a multi-channel beamformer toward a direction-of-arrival (DOA) of the audio signal based at least in part on the directions of the N target faces.
2 . The method of claim 1 , wherein N=1.
3 . The method of claim 2 , wherein the selecting of the N target faces comprises:
determining which of the one or more faces is closest to the plurality of microphones, the face that is closest to the plurality of microphones being selected as the target face.
4 . The method of claim 1 , wherein N>1.
5 . The method of claim 4 , wherein each of the one or more faces is selected as a respective one of the N target faces.
6 . The method of claim 1 , further comprising:
determining, for each of the N target faces, whether the direction of the target face is within a threshold range of the DOA of the audio signal.
7 . The method of claim 6 , wherein the selective steering of the beam comprises:
steering the beam toward the DOA of the audio signal if the direction of at least one of the N target faces is within the threshold range of the DOA of the audio signal.
8 . The method of claim 6 , wherein the selective steering of the beam comprises:
refraining from steering the beam toward the DOA of the audio signal if none of the directions of the N target faces is within the threshold range of the DOA of the audio signal.
9 . The method of claim 6 , wherein the multi-channel beamformer comprises a minimum variance distortionless response (MVDR) beamformer that reduces a power of a noise component of the audio signal without distorting a speech component of the audio signal.
10 . The method of claim 9 , further comprising:
calculating a filter associated with the MVDR beamformer based on a covariance of the noise component of the audio signal and a covariance of the speech component of the audio signal.
11 . The method of claim 10 , further comprising:
determining the covariance of the speech component of the audio signal based on determining that the direction of at least one of the N target faces is within the threshold range of the DOA of the audio signal.
12 . The method of claim 10 , further comprising:
determining the covariance of the noise component of the audio signal based on determining that none of the directions of the N target faces is within the threshold range of the DOA of the audio signal.
13 . A speech enhancement system comprising:
a processing system; and
a memory storing instructions that, when executed by the processing system, causes the speech enhancement system to:
receive an audio signal via a plurality of microphones;
receive an image associated with a frame of the audio signal;
detect one or more faces in the received image;
select a number (N) of target faces among the one or more faces detected in the received image;
determine a respective direction of each of the N target faces relative to the plurality of microphones; and
selectively steer a beam associated with a multi-channel beamformer toward a direction-of-arrival (DOA) of the audio signal based at least in part on the directions of the N target faces.
14 . The speech enhancement system of claim 13 , wherein N=1 and the selecting of the N target faces comprises:
determining which of the one or more faces is closest to the plurality of microphones, the face that is closest to the plurality of microphones being selected as the target face.
15 . The speech enhancement system of claim 13 , wherein N>1 and each of the one or more faces is selected as a respective one of the N target faces.
16 . The speech enhancement system of claim 13 , wherein execution of the instructions further causes the speech enhancement system to:
determine, for each of the N target faces, whether the direction of the target face is within a threshold range of the DOA of the audio signal.
17 . The speech enhancement system of claim 16 , wherein the selective steering of the beam comprises:
steering the beam toward the DOA of the audio signal if the direction of at least one of the N target faces is within the threshold range of the DOA of the audio signal; and
refraining from steering the beam toward the DOA of the audio signal if none of the directions of the N target faces is within the threshold range of the DOA of the audio signal.
18 . The speech enhancement system of claim 16 , wherein the multi-channel beamformer comprises a minimum variance distortionless response (MVDR) beamformer that reduces a power of a noise component of the audio signal without distorting a speech component of the audio signal.
19 . The speech enhancement system of claim 18 , wherein execution of the instructions further causes the speech enhancement system to:
calculate a filter associated with the MVDR beamformer based on a covariance of the noise component of the audio signal and a covariance of the speech component of the audio signal.
20 . The speech enhancement system of claim 19 , wherein execution of the instructions further causes the speech enhancement system to:
determine the covariance of the speech component of the audio signal based on determining that the direction of at least one of the N target faces is within the threshold range of the DOA of the audio signal; and
determine the covariance of the noise component of the audio signal based on determining that none of the directions of the N target faces is within the threshold range of the DOA of the audio signal.