Adaptive multichannel dereverberation for automatic speech recognition
Utilizing an adaptive multichannel technique to mitigate reverberation present in received audio signals, prior to providing corresponding audio data to one or more additional component(s), such as automatic speech recognition (ASR) components. Implementations disclosed herein are “adaptive”, in that they utilize a filter, in the reverberation mitigation, that is online, causal and varies depending on characteristics of the input. Implementations disclosed herein are “multichannel”, in that a corresponding audio signal is received from each of multiple audio transducers (also referred to herein as “microphones”) of a client device, and the multiple audio signals (e.g., frequency domain representations thereof) are utilized in updating of the filter—and dereverberation occurs for audio data corresponding to each of the audio signals (e.g., frequency domain representations thereof) prior to the audio data being provided to ASR component(s) and/or other component(s).
1. A method implemented by one or more processors, comprising:
receiving a plurality of audio signal streams, wherein each of the audio signal streams is based on output from a corresponding one of a plurality of microphones of a client device;
at each of a plurality of iterations during a spoken utterance of a user that is detected at the plurality of microphones and that influences the audio signal streams:
converting most recent unprocessed portions of the audio signal streams into corresponding frequency domain representations;
updating a multi-microphone adaptive reverberation filter utilizing the corresponding frequency domain representations of the audio signal streams for at least one prior iteration of the plurality of iterations, wherein, at a given iteration, updating the multi-microphone adaptive reverberation filter utilizing the corresponding frequency domain representations of the audio signal streams for at least one prior iteration of the plurality of iterations comprises:
updating the multi-microphone adaptive reverberation filter utilizing the corresponding frequency domain representations of the audio signal streams for a prior iteration that is at least N iterations prior to the given iteration, and wherein N is greater than one;
utilizing the updated multi-microphone adaptive reverberation filter in generating reverberation mitigated versions of the corresponding frequency domain representations for the most recent unprocessed portions of the audio signal streams; and
providing the reverberation mitigated versions of the corresponding frequency domain representations for further processing by at least one additional component.
2. The method of claim 1 , wherein converting the most recent unprocessed portions of the audio signal streams into corresponding frequency domain representations comprises applying a fast Fourier transform to each of the unprocessed portions of the audio signal streams to generate the corresponding frequency domain representations of the most recent unprocessed portions of the audio signal streams.
3. The method of claim 2 , wherein the corresponding frequency domain representations each comprise corresponding values for a plurality of frequency bins.
4. The method of claim 1 , wherein the updated multi-microphone adaptive reverberation filter and the corresponding frequency domain representations for the most recent unprocessed portions of the audio signal streams are each a corresponding matrix, and wherein utilizing the updated multi-microphone adaptive reverberation filter in generating the reverberation mitigated versions of the corresponding frequency domain representations for the most recent unprocessed portions of the audio signal streams comprises:
generating a conjugate transpose of the multi-microphone adaptive reverberation filter; and
generating the reverberation mitigated version of the corresponding frequency domain representations for the most recent unprocessed portions of the audio signal streams based on subtracting, from the corresponding frequency domain representations, a given matrix that is based on the conjugate transpose of the multi-microphone adaptive reverberation filter.
5. The method of claim 1 , wherein the further processing comprises performing automatic speech recognition and the additional component comprises an automatic speech recognition component.
6. The method of claim 5 , wherein in performing the automatic speech recognition, the automatic speech recognition component utilizes the trained acoustic model in processing the reverberation mitigated versions of the corresponding frequency domain representations, for a plurality of the iterations, to generate a semantic representation of the reverberation mitigated versions of the corresponding frequency domain representations, for a plurality of the iterations.
7. The method of claim 6 , wherein the trained acoustic model is trained based at least in part on training audio data that is not dereverberated.
8. The method of claim 6 , wherein the trained acoustic model is trained based at least in part on training audio data that is dereverberated.
9. A client device, comprising:
a plurality of microphones; and
one or more processors configured to:
receiving a plurality of audio signal streams, wherein each of the audio signal streams is based on output from a corresponding one of the plurality of microphones;
at each of a plurality of iterations during a spoken utterance of a user that is detected at the plurality of microphones and that influences the audio signal streams:
convert most recent unprocessed portions of the audio signal streams into corresponding frequency domain representations;
update a multi-microphone adaptive reverberation filter utilizing the corresponding frequency domain representations of the audio signal streams for at least one prior iteration of the plurality of iterations, wherein, at a given iteration, in updating the multi-microphone adaptive reverberation filter utilizing the corresponding frequency domain representations of the audio signal streams for at least one prior iteration of the plurality of iterations, one or more of the processors are to:
update the multi-microphone adaptive reverberation filter utilizing the corresponding frequency domain representations of the audio signal streams for a prior iteration that is at least N iterations prior to the given iteration, and wherein N is greater than one;
utilize the updated multi-microphone adaptive reverberation filter in generating reverberation mitigated versions of the corresponding frequency domain representations for the most recent unprocessed portions of the audio signal streams; and
provide the reverberation mitigated versions of the corresponding frequency domain representations for further processing by at least one additional component.
10. The client device of claim 9 , wherein in converting the most recent unprocessed portions of the audio signal streams into corresponding frequency domain representations one or more of the processors are to apply a fast Fourier transform to each of the unprocessed portions of the audio signal streams to generate the corresponding frequency domain representations of the most recent unprocessed portions of the audio signal streams.
11. The client device of claim 10 , wherein the corresponding frequency domain representations each comprise corresponding values for a plurality of frequency bins.
12. The client device of claim 9 , wherein the updated multi-microphone adaptive reverberation filter and the corresponding frequency domain representations for the most recent unprocessed portions of the audio signal streams are each a corresponding matrix, and wherein in utilizing the updated multi-microphone adaptive reverberation filter in generating the reverberation mitigated versions of the corresponding frequency domain representations for the most recent unprocessed portions of the audio signal streams, one or more of the processors are to:
generate a conjugate transpose of the multi-microphone adaptive reverberation filter; and
generate the reverberation mitigated version of the corresponding frequency domain representations for the most recent unprocessed portions of the audio signal streams based on subtracting, from the corresponding frequency domain representations, a given matrix that is based on the conjugate transpose of the multi-microphone adaptive reverberation filter.
13. The client device of claim 9 , wherein the further processing comprises performing automatic speech recognition and the additional component comprises an automatic speech recognition component.
14. The client device of claim 13 , wherein in performing the automatic speech recognition, the automatic speech recognition component utilizes the trained acoustic model in processing the reverberation mitigated versions of the corresponding frequency domain representations, for a plurality of the iterations, to generate a semantic representation of the reverberation mitigated versions of the corresponding frequency domain representations, for a plurality of the iterations.
15. The client device of claim 14 , wherein the trained acoustic model is trained based at least in part on training audio data that is not dereverberated.
16. The client device of claim 14 , wherein the trained acoustic model is trained based at least in part on training audio data that is dereverberated.