Spatio-temporal beamformer
This disclosure provides methods, devices, and systems for signal processing. The present implementations relate more specifically to a spatio-temporal beamformer. In some aspects, a beamforming system may receive an audio signal via a plurality of microphones, the audio signal including a number (B) of frames for each of the plurality of microphones, each of the B frames for each of the plurality of microphones including a number (N) of time-domain samples. For a first microphone, the beamforming system may transform the B*N time-domain samples into B*N/2 first frequency-domain samples; transform the B*N/2 first frequency-domain samples into B*N/2 second frequency-domain samples; and determine a probability of speech associated with the B*N/2 second frequency-domain samples based on a neural network model. The beamformer system may determine a minimum variance distortionless response (MVDR) beamforming filter based at least in part on the probability of speech for the first microphone.
1. A method of processing an audio signal, comprising:
receiving a first audio signal via a plurality of microphones, the first audio signal including a number (B) of frames for each of the plurality of microphones, each of the B frames for each of the plurality of microphones including a number (N) of time-domain samples;
for a first microphone included in the plurality of microphones:
transforming the B*N time-domain samples into B*N/2 first frequency-domain samples based on an N-point fast Fourier transform (FFT);
transforming the B*N/2 first frequency-domain samples into B*N/2 second frequency-domain samples based on a B-point FFT; and
determining a probability of speech associated with the B*N/2 second frequency-domain samples based on a neural network model;
determining a minimum variance distortionless response (MVDR) beamforming filter based at least in part on the probability of speech for the first microphone; and
processing the first audio signal based on the MVDR beamforming filter.
2. The method of claim 1 , further comprising:
generating a first speech signal based on the probability of speech for the first microphone and the B*N/2 second frequency-domain samples;
transforming the first speech signal into a second speech signal based on a B-point inverse FFT;
transforming the second speech signal into a third speech signal, wherein the third speech signal includes a first number of frequency-domain samples associated with a first frequency bin and a second number of frequency-domain samples associated with a second frequency bin, wherein the first and second numbers are different; and,
determining a probability of speech associated with the third speech signal.
3. The method of claim 2 , wherein the determining of the MVDR beamforming filter comprises determining the MVDR beamforming filter based on the probability of speech associated with the third speech signal.
4. The method of claim 3 , further comprising generating a second audio signal based on the B*N/2 first frequency-domain samples, wherein the second audio signal includes the first number of frequency-domain samples associated with the first frequency bin and the second number of frequency-domain samples associated with the second frequency bin.
5. The method of claim 4 , further comprising generating a reconstructed probability of speech based on the probability of speech associated with the B*N/2 second frequency-domain samples.
6. The method of claim 5 , wherein the reconstructed probability of speech comprises:
for the first frequency bin in the probability of speech associated with the B*N/2 second frequency-domain samples:
a first plurality of probability values included in the probability of speech associated with the B*N/2 second frequency-domain samples and corresponding to a first plurality of second frequency-domain samples associated with the first frequency bin;
a second plurality of probability values included in the probability of speech associated with the B*N/2 second frequency-domain samples and corresponding to a second plurality of second frequency-domain samples associated with a third frequency bin preceding the first frequency bin; and
a third plurality of probability values included in the probability of speech associated with the B*N/2 second frequency-domain samples and corresponding to a third plurality of second frequency-domain samples associated with a fourth frequency bin succeeding the first frequency bin.
7. The method of claim 6 , wherein each of the second plurality of probability values is weighted by a respective first weight, and each of the third plurality of probability values is weighted by a respective second weight.
8. The method of claim 1 , wherein the transforming of the B*N time-domain samples into the B*N/2 first frequency-domain samples comprises:
buffering the B frames; and
applying the N-point FFT to the buffered frames.
9. The method of claim 1 , wherein the determining of the probability of speech associated with the B*N/2 second frequency-domain samples comprises decimating the B*N/2 second frequency-domain samples by a decimation factor (D), the probability of speech associated with the B*N/2 second frequency-domain samples being determined based on the B*N/2D decimated second frequency-domain samples.
10. The method of claim 9 , wherein D=2.
11. The method of claim 9 , wherein the decimating of the B*N/2 second frequency-domain samples comprises:
retaining B/2D second frequency-domain samples associated with a first frequency bin; and
discarding B/2D second frequency-domain samples associated with the first frequency bin.
12. The method of claim 1 , further comprising:
determining an average probability of speech for each frequency bin associated with the B*N/2 second frequency-domain samples; and
determining a probability of speech associated with the B*N/2 first frequency-domain samples based on the average probabilities of speech.
13. A beamforming system, comprising:
a processing system; and
a memory storing instructions that, when executed by the processing system, causes the speech enhancement system to:
receive a first audio signal via a plurality of microphones, the first audio signal including a number (B) of frames for each of the plurality of microphones, each of the B frames for each of the plurality of microphones including a number (N) of time-domain samples;
for a first microphone included in the plurality of microphones:
transform the B*N time-domain samples into B*N/2 first frequency-domain samples based on an N-point fast Fourier transform (FFT);
transform the B*N/2 first frequency-domain samples into B*N/2 second frequency-domain samples based on a B-point FFT; and
determine a probability of speech associated with the B*N/2 second frequency-domain samples based on a neural network model;
determine a minimum variance distortionless response (MVDR) beamforming filter based at least in part on the probability of speech for the first microphone; and
process the first audio signal based on the MVDR beamforming filter.
14. The beamforming system of claim 13 , wherein execution of the instructions further causes the beamforming system to:
generate a first speech signal based on the probability of speech for the first microphone and the B*N/2 second frequency-domain samples;
transform the first speech signal into a second speech signal based on a B-point inverse FFT;
transform the second speech signal into a third speech signal, wherein the third speech signal includes a first number of frequency-domain samples associated with a first frequency bin and a second number of frequency-domain samples associated with a second frequency bin, wherein the first and second numbers are different; and,
determine a probability of speech associated with the third speech signal.
15. The beamforming system of claim 14 , wherein execution of the instructions further causes the beamforming system to determine the MVDR beamforming filter based on the probability of speech associated with the third speech signal.
16. The beamforming system of claim 15 , wherein execution of the instructions further causes the beamforming system to generate a second audio signal based on the B*N/2 first frequency-domain samples, wherein the second audio signal includes the first number of frequency-domain samples associated with the first frequency bin and the second number of frequency-domain samples associated with the second frequency bin.
17. The beamforming system of claim 16 , wherein execution of the instructions further causes the beamforming system to generate a reconstructed probability of speech based on the probability of speech associated with the B*N/2 second frequency-domain samples.
18. The beamforming system of claim 17 , wherein the reconstructed probability of speech comprises:
for the first frequency bin in the probability of speech associated with the B*N/2 second frequency-domain samples:
a first plurality of probability values included in the probability of speech associated with the B*N/2 second frequency-domain samples and corresponding to a first plurality of second frequency-domain samples associated with the first frequency bin;
a second plurality of probability values included in the probability of speech associated with the B*N/2 second frequency-domain samples and corresponding to a second plurality of second frequency-domain samples associated with a third frequency bin preceding the first frequency bin; and
a third plurality of probability values included in the probability of speech associated with the B*N/2 second frequency-domain samples and corresponding to a third plurality of second frequency-domain samples associated with a fourth frequency bin succeeding the first frequency bin.
19. The beamforming system of claim 13 , wherein execution of the instructions further causes the beamforming system to:
buffer the B frames; and
apply the N-point FFT to the buffered frames.
20. The beamforming system of claim 13 , wherein execution of the instructions further causes the beamforming system to decimate the B*N/2 second frequency-domain samples by a decimation factor (D), the probability of speech associated with the B*N/2 second frequency-domain samples being determined based on the B*N/2D decimated second frequency-domain samples.