Audio device with distractor suppression
An audio device is disclosed, comprising multiple microphones and an audio module. The multiple microphones generate multiple audio signals. The audio module coupled to the multiple microphones comprises a processor, a storage media and a post-processing circuit. The storage media includes instructions operable to be executed by the processor to perform operations comprising: producing multiple instantaneous relative transfer functions (IRTFs) using a known adaptive algorithm according to multiple spectral representations for multiple first sample values in current frames of the multiple audio signals; and, performing distractor suppression over the multiple spectral representations and the multiple IRTFs using an end-to-end neural network to generate a compensation mask. The post-processing circuit generates an audio output signal according to the compensation mask. Each IRTF represents a difference in sound propagation between each predefined microphone and a reference microphone of the microphones relative to sound sources.
1 . An audio device, comprising:
multiple microphones that generate multiple audio signals; and
an audio module coupled to the multiple microphones, comprising:
at least one processor;
at least one storage media including instructions operable to be executed by the at least one processor to perform a set of operations comprising:
producing multiple instantaneous relative transfer functions (IRTFs) using a first known adaptive algorithm according to multiple mic spectral representations for multiple first sample values in current frames of the multiple audio signals; and
performing distractor suppression over the multiple mic spectral representations and the multiple IRTFs using an end-to-end neural network to generate a compensation mask; and
a post-processing circuit that generates an audio output signal according to the compensation mask;
wherein each IRTF represents a difference in sound propagation between each predefined microphone and a reference microphone of the multiple microphones relative to at least one sound source; and
wherein each predefined microphone is different from the reference microphone.
2 . The audio device according to claim 1 , further comprising:
a loudspeaker that converts a playback audio signal into a sound pressure signal;
wherein the set of operations further comprises:
producing multiple playback transfer functions (PTFs) using a second known adaptive algorithm according to the multiple mic spectral representations and a playback spectral representation for multiple second sample values in a current frame of the playback audio signal; and
performing acoustic echo cancellation (AEC) over the multiple mic spectral representations, the multiple IRTFs, the playback spectral representation and the multiple PTFs using the end-to-end neural network to generate the compensation mask;
wherein each PTF indicates a degree of a sound leakage from the loudspeaker to a target microphone of the multiple microphones.
3 . The audio device according to claim 2 , wherein the first and the second known adaptive algorithms are least mean square (LMS) algorithm.
4 . The audio device according to claim 2 , wherein each PTF includes multiple PTF elements corresponding to multiple frequency bands, and wherein the operation of producing the PTFs comprises:
for a target frequency band of one PTF,
producing a current PTF element for the one PTF using the second known adaptive algorithm according to a first corresponding sample in the playback spectral representation and a difference between an estimated sample and a second corresponding sample in a corresponding mic spectral representation for the target microphone;
wherein the estimated sample is related to a product of a previous PTF element for the one PTF and the first corresponding sample.
5 . The audio device according to claim 2 , wherein the set of operations further comprises:
performing active noise cancellation (ANC) operations over the multiple first sample values using the end-to-end neural network to generate multiple third sample values.
6 . The audio device according to claim 5 , wherein the end-to-end neural network comprises:
a time delay neural network (TDNN);
a first long short-term memory (LSTM) network coupled to the output of the TDNN; and
a second LSTM network coupled to the output of the TDNN;
wherein the TDNN and the first LSTM network are jointly trained to perform the ANC operations over the first sample values to generate the third sample values;
wherein the TDNN and the second LSTM network are jointly trained to perform the distraction suppression over the multiple mic spectral representations and the multiple IRTFs to generate the compensation mask; and
wherein the TDNN and the second LSTM network are jointly trained to perform the AEC over the multiple mic spectral representations, the multiple IRTFs, the playback spectral representation and the multiple PTFs to generate the compensation mask.
7 . The audio device according to claim 5 , wherein the post-processing circuit modifies a main spectral representation of the multiple mic spectral representations with the compensation mask to generate a compensated spectral representation, and generates the audio output signal according to the multiple third sample values and the compensated spectral representation.
8 . The audio device according to claim 1 , wherein the compensation mask comprises multiple frequency band gains, each indicating its corresponding frequency band is either speech-dominant or noise-dominant.
9 . The audio device according to claim 1 , wherein the end-to-end neural network is a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a time delay neural network (TDNN) or a combination thereof.
10 . The audio device according to claim 1 , wherein one of the multiple microphones is selected as the reference microphone according to signal-to-noise ratios or receiving spectrum ranges of the multiple microphones.
11 . The audio device according to claim 1 , wherein the audio output signal is sent out over a connection link.
12 . The audio device according to claim 1 , further comprising:
an audio output circuit that is coupled to an output terminal of the post-processing circuit over a connection link and converts the audio output signal into a sound pressure signal.
13 . The audio device according to claim 1 , wherein the audio module and at least one of the multiple microphones are arranged at a first source device, wherein the other microphones are arranged at a second source device, and wherein the audio module connected to the at least one microphone is coupled to the other microphones over a first connection link and to a sink device over a second connection link.
14 . The audio device according to claim 1 , wherein the multiple microphones are respectively arranged at two different source devices, and wherein the audio module is arranged at a sink device and coupled to the multiple microphones over two different connection links.
15 . The audio device according to claim 1 , wherein the multiple microphones are respectively arranged at a first source device, a second source device and a sink device, and the audio module is arranged at the sink device, and wherein the audio module is coupled to the multiple microphones over three different connection links.
16 . The audio device according to claim 1 , wherein each IRTF includes multiple IRTF elements corresponding to multiple frequency bands, and wherein the operation of producing the IRTFs comprises:
for a target frequency band of one IRTF,
producing a current IRTF element for the one IRTF using the first known adaptive algorithm according to a first corresponding sample in a first mic spectral representation for the predefined microphone and a difference between an estimated sample and a second corresponding sample in a second mic spectral representation for the reference microphone;
wherein the estimated sample is related to a product of a previous IRTF element for the one IRTF and the first corresponding sample.
17 . An audio apparatus, comprising:
two audio devices of claim 1 that are arranged at two different source devices;
wherein the two audio output signals from the two audio devices are respectively sent to a sink device over a first connection link and a second connection link.
18 . The apparatus according to claim 17 , further comprising:
an audio output circuit that is arranged at the sink device and converts the two audio output signals into two sound pressure signals.
19 . The audio apparatus according to claim 17 , wherein the sink device receives the two audio output signals over the first and the second connection links and delivers them over a third connection link.
20 . An audio processing method, comprising:
obtaining multiple instantaneous relative transfer functions (IRTFs) using a first known adaptive algorithm according to multiple mic spectral representations for multiple first sample values in current frames of multiple audio signals from multiple microphones;
performing distractor suppression over the multiple mic spectral representations and the multiple IRTFs using an end-to-end neural network to generate a compensation mask; and
obtaining an audio output signal according to the compensation mask;
wherein each IRTF represents a difference in sound propagation between each predefined microphone and a reference microphone of the multiple microphones relative to at least one sound source; and
wherein each predefined microphone is different from the reference microphone.
21 . The method according to claim 20 , further comprising:
producing multiple playback transfer functions (PTFs) using a second known adaptive algorithm according to the multiple mic spectral representations and a playback spectral representation for multiple second sample values in a current frame of a playback audio signal for a loudspeaker; and
performing acoustic echo cancellation (AEC) over the multiple mic spectral representations, the multiple IRTFs, the playback spectral representation and the multiple PTFs using the end-to-end neural network to generate the compensation mask;
wherein each PTF indicates a degree of a sound leakage from the loudspeaker to a target microphone of the multiple microphones.
22 . The method according to claim 21 , wherein the first and the second known adaptive algorithms are least mean square (LMS) algorithm.
23 . The method according to claim 21 , wherein each PTF includes multiple PTF elements corresponding to multiple frequency bands, wherein the step of obtaining the PTFs comprises:
for a target frequency band of one PTF,
producing a current PTF element for the one PTF using the second known adaptive algorithm according to a first corresponding sample in the playback spectral representation and a difference between an estimated sample and a second corresponding sample in a corresponding mic spectral representation for the target microphone;
wherein the estimated sample is related to a product of a previous PTF element for the one PTF and the first corresponding sample.
24 . The method according to claim 21 , further comprising:
performing active noise cancellation (ANC) operations over the multiple first sample values using the end-to-end neural network to generate multiple third sample values.
25 . The method according to claim 24 , wherein the end-to-end neural network comprises a time delay neural network (TDNN), a first long short-term memory (LSTM) network and a second LSTM network, wherein the TDNN and the first LSTM network are jointly trained to perform the ANC operations over the first sample values to generate the third sample values, wherein the TDNN and the second LSTM network are jointly trained to perform the distraction suppression over the multiple mic spectral representations and the multiple IRTFs to generate the compensation mask, and wherein the TDNN and the second LSTM network are jointly trained to perform the AEC over the multiple mic spectral representations, the multiple IRTFs, the playback spectral representation and the multiple PTFs to generate the compensation mask.
26 . The method according to claim 24 , wherein the step of obtaining the audio output signal comprises:
modifying a main spectral representation of the multiple mic spectral representations with the compensation mask to obtain a compensated spectral representation; and
obtaining the audio output signal according to the third sample values and the compensated spectral representation.
27 . The method according to claim 20 , wherein the compensation mask comprises multiple frequency band gains, each indicating its corresponding frequency band is either speech-dominant or noise-dominant.
28 . The method according to claim 20 , further comprising:
selecting one of the multiple microphones as the reference microphone according to either signal-to-noise ratios or receiving spectrum ranges of the multiple microphones.
29 . The method according to claim 20 , further comprising:
sending out the audio output signal over a connection link.
30 . The method according to claim 20 , further comprising:
converting the audio output signal into a sound pressure signal.
31 . The method according to claim 20 , further comprising:
at a first source device with M microphones of the multiple microphones,
carrying out the above steps of obtaining the multiple IRTFs, performing and obtaining the audio output signal based on M audio signals of the M microphones;
defining the audio output signal in the first source device as a first signal; and
delivering the first signal to a sink device over a first connection link;
at a second source device with N microphones of the multiple microphones,
carrying out the above steps of obtaining the multiple IRTFs, performing and obtaining the audio output signal based on N audio signals of the N microphones;
defining the audio output signal in the second source device as a second signal; and
delivering the second signal to the sink device over a second connection link; and
at the sink device,
receiving the first and the second signals over the first and the second connection links, where M, N>1.
32 . The method according to claim 20 , further comprising:
at a first source device with at least one of the multiple microphones,
delivering at least one audio signal from the at least one of the multiple microphones to a second source device with the other microphones over a first connection link;
at the second source device,
carrying out the above steps of obtaining the multiple IRTFs, performing and obtaining the audio output signal according to the multiple audio signals; and
delivering the audio output signal to a sink device over a second connection link; and
at the sink device,
receiving the audio output signal over the second connection link; and
transmitting the audio output signal over a third connection link.
33 . The method according to claim 20 , further comprising:
at a first source device with at least one of the multiple microphones,
delivering at least one audio signal from the at least one of the multiple microphones to a sink device over a first connection link;
at a second source device with the other microphones,
delivering the other audio signals from the other microphones to the sink device over a second connection link; and
at the sink device,
carrying out the above steps of obtaining the multiple IRTFs, performing and obtaining the audio output signal according to the multiple audio signals; and
delivering the audio output signal over a third connection link.
34 . The method according to claim 20 , further comprising:
at a first source device with a first portion of the multiple microphones,
delivering at least one audio signal from the first portion of the multiple microphones to a sink device over a first connection link;
at a second source device with a second portion of the multiple microphones,
delivering at least one audio signal from the second portion of the multiple microphones to the sink device over a second connection link;
at the sink device with a third portion of the multiple microphones,
performing the above steps of obtaining the multiple IRTFs, performing and obtaining the audio output signal according to the multiple audio signals of the multiple microphones; and
delivering the audio output signal over a third connection link;
wherein the multiple microphones are divided into the first, the second and the third portions.
35 . The method according to claim 20 , wherein each IRTF includes multiple IRTF elements corresponding to multiple frequency bands, wherein the step of obtaining the IRTFs comprises:
for a target frequency band of one IRTF,
obtaining a current IRTF element for the one IRTF using the first known adaptive algorithm according to a first corresponding sample in a first mic spectral representation for the predefined microphone and a difference between an estimated sample and a second corresponding sample in a second mic spectral representation for the reference microphone;
wherein the estimated sample is related to a product of a previous IRTF element for the one IRTF and the first corresponding sample.