Adaptive audio enhancement for multichannel speech recognition

1. A computer-implemented method comprising:

receiving a first channel of audio data corresponding to an utterance and a second channel of audio data corresponding to the utterance;

generating, using a first neural network, (i) a first set of filter parameters for a first filter and (ii) a second set of filter parameters for a second filter;

generating a single combined channel of audio data by combining (i) audio data of the first channel that has been filtered using the first filter and (ii) audio data of the second channel that has been filtered using the second filter;

inputting the audio data for the single combined channel to a second neural network trained as a speech recognition model; and

providing a transcription for the utterance that is determined based at least on output that the second neural network provides in response to receiving the audio data for the single combined channel.

2. The method of claim 1 , wherein the first neural network comprises one or more long short-term memory layers.

3. The method of claim 1 , wherein the first neural network comprises:

a first long short-term memory layer that receives both first and second channels of audio; and

a second long short-term memory layer corresponding to the first channel and a third long short-term memory layer corresponding to the second channel, the second long short-term memory layer and the third long short-term memory layer each receiving the output of the first long short-term memory layer and providing a set of filter parameters for the corresponding channel.

4. The method of claim 3 , wherein the long short-term memory layer layers have parameters that have been learned during a training process that jointly trains the long short-term memory layers and the second neural network that is trained as a speech recognition model.

5. The method of claim 1 , comprising:

changing, or generating, new filter parameters for each input frame of audio data.

6. The method of claim 1 , comprising:

for each audio frame in a sequence of audio frames of the utterance, generating and a new set of filter parameters and convolving audio data for the frame with a filter with the new set of filter parameters.

7. The method of claim 1 , wherein the first filter and the second filter are finite impulse response filters.

8. The method of claim 1 , wherein the first filter and the second filter have different parameters.

9. The method of claim 1 , wherein different microphone outputs are convolved with different filters.

10. The method of claim 1 , wherein the first and second channels of audio data are first and second channels of audio waveform data for the utterance,

wherein the first and second channels of audio waveform are recordings of the utterance by different microphones that are spaced apart from each other.

11. The method of claim 1 , wherein the second neural network trained as a speech recognition model comprises a convolutional layer, one or more long-short term memory layers, and multiple hidden layers.

12. The method of claim 11 , wherein the convolutional layer of the second neural network trained as a speech recognition model is configured to perform a time domain convolution.

13. The method of claim 11 , wherein the second neural network trained as a speech recognition model is configured such that output of the convolutional layer is pooled to generate a set of pooled values.

14. The method of claim 13 , wherein the second neural network trained as a speech recognition model is configured to input the pooled values to one or more long-short term memory layers within the second neural network trained as a speech recognition model.

15. The method of claim 1 , wherein the first and second filters are configured to perform both spatial and spectral filtering.

16. The method of claim 1 , comprising:

convolving the audio data for the first channel with a first filter having the first set of filter parameters to generate first convolution outputs;

convolving the audio data for the second channel with a second filter having the second set of filter parameters to generate second convolution outputs; and

combining the first convolution outputs and the second convolution outputs.

17. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

receiving a first channel of audio data corresponding to an utterance and a second channel of audio data corresponding to the utterance;

generating, using a first neural network, (i) a first set of filter parameters for a first filter and (ii) a second set of filter parameters for a second filter;

generating a single combined channel of audio data by combining (i) audio data of the first channel that has been filtered using the first filter and (ii) audio data of the second channel that has been filtered using the second filter;

inputting the audio data for the single combined channel to a second neural network trained as a speech recognition model; and

providing a transcription for the utterance that is determined based at least on output that the second neural network provides in response to receiving the audio data for the single combined channel.

18. The system of claim 17 , wherein the first and second filters are configured to perform both spatial and spectral filtering.

19. The system of claim 17 , wherein the operations further comprise:

convolving the audio data for the first channel with a first filter having the first set of filter parameters to generate first convolution outputs;

convolving the audio data for the second channel with a second filter having the second set of filter parameters to generate second convolution outputs; and

combining the first convolution outputs and the second convolution outputs.

20. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

receiving a first channel of audio data corresponding to an utterance and a second channel of audio data corresponding to the utterance;

generating, using a first neural network, (i) a first set of filter parameters for a first filter and (ii) a second set of filter parameters for a second filter;

generating a single combined channel of audio data by combining (i) audio data of the first channel that has been filtered using the first filter and (ii) audio data of the second channel that has been filtered using the second filter;

inputting the audio data for the single combined channel to a second neural network trained as a speech recognition model; and

providing a transcription for the utterance that is determined based at least on output that the second neural network provides in response to receiving the audio data for the single combined channel.

21. The method of claim 1 , wherein:

the first filter is a first time-domain filter,

the second filter is a second time-domain filter, and

generating the single combined channel of audio data comprises:

performing a time-domain convolution using the first filter to generate the audio data of the first channel that has been filtered using the first filter; and

performing a time-domain convolution using the second filter to generate the audio data of the second channel that has been filtered using the second filter.

22. The method of claim 1 , wherein the first neural network has parameters that have been learned through a training process that jointly trains the first neural network and the second neural network.