Robust short-time fourier transform acoustic echo cancellation during audio playback
Example techniques involve noise-robust acoustic echo cancellation. An example implementation may involve causing one or more speakers of the playback device to play back audio content and while the audio content is playing back, capturing, via the one or more microphones, audio within an acoustic environment that includes the audio playback. The example implementation may involve determining measured and reference signals in the STFT domain. During each n th iteration of an acoustic echo canceller (AEC): the implementation may involve determining a frame of an output signal by generating a frame of a model signal by passing a frame of the reference signal through an instance of an adaptive filter and then redacting the n th frame of the model signal from an n th frame of the measured signal. The implementation may further involve determining an instance of the adaptive filter for a next iteration of the AEC.
1. A system comprising:
at least one playback device comprising at least one headphone, at least one microphone and one or more audio transducers;
a mobile device;
one or more wireless network interfaces;
at least one processor; and
data storage storing program instructions that are executable by the at least one processor to cause the system to perform functions comprising:
playing back at least one audio signal via the one or more audio transducers of the at least one playback device;
while playing back the at least one audio signal, capturing, via the at least one microphone, audio, wherein at least a portion of the captured audio represents sound produced by the one or more audio transducers in playing back the at least one audio signal;
transforming into a short time Fourier transform (STFT) domain the captured audio to generate a measured signal representing actual acoustic echo;
transforming into the STFT domain the at least one audio signal being played back by the at least one playback device to generate a reference signal;
during each n th iteration of an acoustic echo canceller (AEC):
determining an n th frame of an output signal, wherein determining the n th frame of the output signal comprises:
(i) generating an n th frame of a model signal representing estimated acoustic echo by passing an n th frame of the reference signal through an n th instance of an adaptive filter; and
(ii) generating the n th frame of the output signal by differencing the n th frame of the model signal and an n th frame of the measured signal;
determining a n+1 th instance of the adaptive filter for a next iteration of the AEC, wherein determining the n+1 th instance of the adaptive filter for the next iteration of the AEC comprises:
(i) estimating an n th frame of an error signal, the n th frame of the error signal representing a difference between the n th frame of the measured signal and the n th frame of the model signal;
(ii) converting the n th frame of an error signal to an n th update filter;
(iii) deactivating inactive portions of the n th update filter, the inactive portions having less than a threshold energy; and
(iv) generating the n+1 th instance of the adaptive filter for the next iteration of the AEC by summing the n th instance of the adaptive filter with the n th update filter; and
sending, via the one or more wireless network interfaces, the output signal as a voice input to one or more voice assistants for processing of the voice input.
2. The system of claim 1 , wherein the at least one headphone comprises wireless headphones, wherein the one or more audio transducers comprise a first audio transducer and a second audio transducer of the wireless headphones, and wherein the at least one audio signal comprises a first audio channel and a second audio channel, and wherein playing back the at least one audio signal comprises:
playing back the first audio channel via the first audio transducer; and
playing back the second audio channel via the second audio transducer.
3. The system of claim 1 , wherein the at least one playback device comprises a first playback device and a second playback device configured in a bonded configuration, wherein the one or more audio transducers comprise a first audio transducer and a second audio transducer, and wherein the at least one audio signal comprises a first audio channel and a second audio channel, and wherein playing back at least one audio signal comprises:
playing back the first audio channel via the first audio transducer; and
playing back the second audio channel via the second audio transducer.
4. The system of claim 1 , wherein the mobile device comprises the AEC, and wherein the functions further comprise:
sending, via the one or more wireless network interfaces to the mobile device, data representing the captured audio.
5. The system of claim 1 , wherein the one or more wireless network interfaces comprise an 802.15-compatible wireless network interface, and wherein the functions further comprise:
sending, via the 802.15-compatible wireless network interface to the at least one playback device, data representing the at least one audio signal.
6. The system of claim 1 , wherein converting the n th frame of an error signal to the n th update filter comprises determining a normalized least mean square (NMLS) of the n th frame of the error signal, and wherein deactivating the inactive portions of the n th update filter comprises determining a sparse NMLS of the n th frame of the error signal by applying to the NMLS of the n th frame of the error signal, a sparse partition criterion that zeroes out frequency bands of the NMLS having less than the threshold energy.
7. The system of claim 6 , wherein determining the normalized least mean square (NMLS) of the n th frame of the error signal comprises applying a frequency-dependent regularization parameter to adapt an NMLS learning rate of change between AEC iterations according to a magnitude of the measured signal.
8. The system of claim 6 , wherein converting the sparse NMLS of the n th frame of the error signal to the n th update filter comprises:
converting the sparse NMLS of the n th frame to a matrix of filter coefficients; and
cross-band filtering the matrix of filter coefficients to generate the n th update filter.
9. The system of claim 6 , wherein the data storage further comprises program instructions that are executable by the at least one processor such that the system is configured to perform functions further comprising:
before determining the NMLS of the n th frame of the error signal, applying an error recovery non-linearity function to the error signal to limit the error signal to a threshold magnitude, wherein determining the normalized least mean square (NMLS) of the n th frame of the error signal comprises determining the NMLS of the n th frame of the limited error signal.
10. The system of claim 9 , wherein the error recovery non-linearity function comprises a non-linear clipping function that limits portions of the error signal that are above the threshold magnitude to the threshold magnitude.
11. A mobile device comprising:
one or more wireless network interfaces;
at least one processor; and
data storage storing program instructions that are executable by the at least one processor to cause the mobile device to perform functions comprising:
causing at least one playback device to play back at least one audio signal via one or more audio transducers of the at least one playback device, wherein the at least one playback device comprises at least one headphone;
while the at least one playback device is playing back the at least one audio signal, receiving audio captured via at least one microphone of the at least one playback device, wherein at least a portion of the captured audio represents sound produced by one or more audio transducers of the at least one playback device in playing back the at least one audio signal;
transforming into a short time Fourier transform (STFT) domain the captured audio to generate a measured signal representing actual acoustic echo;
transforming into the STFT domain the at least one audio signal being played back by the at least one playback device to generate a reference signal;
during each n th iteration of an acoustic echo canceller (AEC):
determining an n th frame of an output signal, wherein determining the n th frame of the output signal comprises:
(i) generating an n th frame of a model signal representing estimated acoustic echo by passing an n th frame of the reference signal through an n th instance of an adaptive filter; and
(ii) generating the n th frame of the output signal by differencing the n th frame of the model signal and an n th frame of the measured signal;
determining a n+1 th instance of the adaptive filter for a next iteration of the AEC, wherein determining the n+1 th instance of the adaptive filter for the next iteration of the AEC comprises:
(i) estimating an n th frame of an error signal, the n th frame of the error signal representing a difference between the n th frame of the measured signal and the n th frame of the model signal;
(ii) converting the n th frame of an error signal to an n th update filter;
(iii) deactivating inactive portions of the n th update filter, the inactive portions having less than a threshold energy; and
(iv) generating the n+1 th instance of the adaptive filter for the next iteration of the AEC by summing the n th instance of the adaptive filter with the n th update filter; and
sending, via the one or more wireless network interfaces, the output signal as a voice input to one or more voice assistants for processing of the voice input.
12. The mobile device of claim 11 , wherein the at least one headphone comprise wireless headphones, wherein the one or more audio transducers comprise a first audio transducer and a second audio transducer of the wireless headphones, and wherein causing the at least one playback device to play back the at least one audio signal comprises:
causing the wireless headphones to play back the at least one audio signal via at the first audio transducer and the second audio transducer.
13. The mobile device of claim 11 , wherein the at least one playback device comprises a first playback device and a second playback device configured in a bonded configuration, wherein the one or more audio transducers comprise a first audio transducer and a second audio transducer, and wherein the at least one audio signal comprises a first audio channel and a second audio channel, and wherein playing back the at least one audio signal comprises:
causing the first playback device to play back the first audio channel via the first audio transducer; and
causing the second playback device to play back the second audio channel via the second audio transducer.
14. The mobile device of claim 11 , wherein the one or more wireless network interfaces comprise an 802.15-compatible wireless network interface, and wherein the functions further comprise:
sending, via the 802.15-compatible wireless network interface to the at least one playback device, data representing the at least one audio signal.
15. The mobile device of claim 11 , wherein converting the n th frame of an error signal to the n th update filter comprises determining a normalized least mean square (NMLS) of the n th frame of the error signal, and wherein deactivating inactive portions of the n th update filter comprises determining a sparse NMLS of the n th frame of the error signal by applying to the NMLS of the n th frame of the error signal, a sparse partition criterion that zeroes out frequency bands of the NMLS having less than the threshold energy.
16. The mobile device of claim 15 , wherein determining the normalized least mean square (NMLS) of the n th frame of the error signal comprises applying a frequency-dependent regularization parameter to adapt an NMLS learning rate of change between AEC iterations according to a magnitude of the measured signal.
17. The mobile device of claim 15 , wherein converting the sparse NMLS of the n th frame of the error signal to the n th update filter comprises:
converting the sparse NMLS of the n th frame to a matrix of filter coefficients; and
cross-band filtering the matrix of filter coefficients to generate the n th update filter.
18. The mobile device of claim 15 , wherein the data storage further comprises program instructions that are executable by the at least one processor such that the mobile device is configured to perform functions further comprising:
before determining the NMLS of the n th frame of the error signal, applying an error recovery non-linearity function to the error signal to limit the error signal to a threshold magnitude, wherein determining the normalized least mean square (NMLS) of the n th frame of the error signal comprises determining the NMLS of the n th frame of the limited error signal.
19. The mobile device of claim 18 , wherein the error recovery non-linearity function comprises a non-linear clipping function that limits portions of the error signal that are above the threshold magnitude to the threshold magnitude.
20. At least one non-transitory computer-readable medium comprising program instructions that are executable by at least one processor such that a system is configured to perform functions comprising:
causing at least one playback device to play back at least one audio signal via one or more audio transducers of the at least one playback device, wherein the at least one playback device comprises at least one headphone;
while the at least one playback device is playing back the at least one audio signal, receiving audio captured via at least one microphone of the at least one playback device, wherein at least a portion of the captured audio represents sound produced by the one or more audio transducers of the at least one playback device in playing back the at least one audio signal;
transforming into a short time Fourier transform (STFT) domain the captured audio to generate a measured signal representing actual acoustic echo;
transforming into the STFT domain the at least one audio signal being played back by the at least one playback device to generate a reference signal;
during each n th iteration of an acoustic echo canceller (AEC):
determining an n th frame of an output signal, wherein determining the n th frame of the output signal comprises:
(i) generating an n th frame of a model signal representing estimated acoustic echo by passing an n th frame of the reference signal through an n th instance of an adaptive filter; and
(ii) generating the n th frame of the output signal by differencing the n th frame of the model signal and an n th frame of the measured signal;
determining a n+1 th instance of the adaptive filter for a next iteration of the AEC, wherein determining the n+1 th instance of the adaptive filter for the next iteration of the AEC comprises:
(i) estimating an n th frame of an error signal, the n th frame of the error signal representing a difference between the n th frame of the measured signal and the n th frame of the model signal;
(ii) converting the n th frame of an error signal to an n th update filter;
(iii) deactivating inactive portions of the n th update filter, the inactive portions having less than a threshold energy; and
(iv) generating the n+1 th instance of the adaptive filter for the next iteration of the AEC by summing the n th instance of the adaptive filter with the n th update filter; and
sending, via one or more wireless network interfaces, the output signal as a voice input to one or more voice assistants for processing of the voice input.