Joint acoustic echo cancellation (AEC) and personalized noise suppression (PNS)

1 . A data processing system comprising:

a processor; and

a machine-readable medium storing executable instructions that, when executed, cause the processor to perform operations comprising:

receiving a far-end signal associated with a first computing device participating in an online communication session;

encoding the far-end signal using a first learnable encoder to generate first embeddings representing first features of the far-end signal;

receiving a near-end signal associated with a second computing device participating in the online communication session, the near-end signal including speech of a target speaker, one or more interfering speakers, and an echo signal;

encoding the near-end signal using a second learnable encoder to generate second embeddings representing second features of the near-end signal;

providing the first embeddings representing features of the far-end signal, the second embeddings representing features of the near-end signal, and an indication of the target speaker as an input to a machine learning model comprising:

an alignment block, the alignment block configured to use attention to align the first features of the far-end signal included in the first embeddings and the second features of the near-end signal included in the second embeddings, the alignment block being further configured to output attention weights;

a first concatenation block configured to concatenate the first embeddings, the second embeddings, and the attention weights into a first concatenated input,

a first group of Long Short-Term Memory (LSTM) blocks trained to perform acoustic echo cancellation (AEC) on the first concatenated input to remove an echo and to output third features,

a second concatenation block configured to concatenate the third features output by the first group of LSTM blocks and the attention weights output by the alignment block into a second concatenated input, and

a second group of LSTM blocks trained to perform personalized noise suppression (PNS) on the second concatenated input to remove speech from one or more interfering speakers and to remove a residual echo output by the first group of LSTM blocks; and

obtaining an audio signal comprising the speech of the target speaker from the machine learning model.

2 . The data processing system of claim 1 , wherein the indication of the target speaker comprises a target speaker embedding vector representing speech characteristics of the target speaker.

3 . The data processing system of claim 2 , wherein the machine-readable medium includes instructions configured to cause the processor to perform operations of:

generating the target speaker embedding vector by capturing audio content comprising speech of the target speaker and extracting features from the audio content.

4 . The data processing system of claim 2 , wherein the second group of LSTM blocks receives the target speaker embedding vector as an input.

5 . The data processing system of claim 1 , wherein the machine-readable medium includes instructions configured to cause the processor to perform operations of sending the audio signal to the first computing device.

6 . A method implemented in a data processing system for processing audio signals, the method comprising:

receiving a far-end signal associated with a first computing device participating in an online communication session;

encoding the far-end signal using a first learnable encoder to generate first embeddings representing first features of the far-end signal;

receiving a near-end signal associated with a second computing device participating in the online communication session, the near-end signal including speech of a target speaker, one or more first interfering speakers, and an echo signal;

encoding the near-end signal using a second learnable encoder to generate second embeddings representing second features of the near-end signal;

analyzing the first embeddings representing features of the far-end signal, the second embeddings representing features of the near-end signal, and an indication of the target speaker as an input to a machine learning model by:

aligning the first features of the far-end signal included in the first embeddings and the second features of the near-end signal included in the second embeddings using an alignment block configured to use attention to align the first features and the second features;

obtaining attention weights output by the alignment block in response to aligning the first features and the second features;

concatenating the first embeddings, the second embeddings, and the attention weights into a first concatenated input using a first concatenation block;

performing acoustic echo cancellation (AEC) on the first concatenated input using a first group of Long Short-Term Memory (LSTM) blocks to remove an echo and to output third features,

concatenating the third features output by the first group of LSTM blocks and the attention weights output by the alignment block into a second concatenated input using a second concatenation block,

performing personalized noise suppression (PNS) on the second concatenated input using a second group of LSTM blocks to remove speech from the one or more interfering speakers and to remove a residual echo output by the first group of LSTM blocks, and

outputting an audio signal comprising speech of the target speaker; and

obtaining the audio signal comprising the speech of the target speaker from the machine learning model.

7 . The method of claim 6 , wherein the indication of the target speaker comprises a speaker embedding vector representing speech characteristics of the target speaker.

8 . The method of claim 7 , further comprising generating the speaker embedding vector by capturing audio content comprising speech of the target speaker and extracting features from the audio content.

9 . The method of claim 6 , further comprising performing AEC on features extracted from the near-end signal to remove echoes from the near-end signal using the machine learning model.

10 . The method of claim 9 , further comprising aligning features of the near-end signal and features of the far-end signal using attention using an alignment block.

11 . The method of claim 10 , further comprising providing alignment information output by the alignment block as an input to the first group of LSTM blocks.

12 . A data processing system comprising:

a processor; and

a machine-readable medium storing executable instructions that, when executed, cause the processor to perform operations comprising:

training a machine learning model using a first batch of training data to train the machine learning model to perform acoustic echo cancellation (AEC) to remove echoes from an input audio signal;

training the machine learning model using a second batch of training data to train the machine learning model to perform personalized noise suppression (PNS) to extract speech associated with a target speaker from the input audio signal;

training the machine learning model using a third batch of training data to train the machine learning model to perform both AEC and PNS on the input audio signal; and

analyzing audio signals associated with a communication session using the machine learning model to remove echoes and to extract the speech of a target speaker participating in the communication session by:

receiving a far-end signal associated with a first computing device participating in an online communication session;

encoding the far-end signal using a first learnable encoder to generate first embeddings representing first features of the far-end signal;

receiving a near-end signal associated with a second computing device participating in the online communication session, the near-end signal including speech of a target speaker, one or more interfering speakers, and an echo signal;

encoding the near-end signal using a second learnable encoder to generate second embeddings representing second features of the near-end signal;

analyzing the first embeddings representing features of the far-end signal, the second embeddings representing features of the near-end signal, and an indication of the target speaker as an input to a machine learning model by:

aligning the first features of the far-end signal included in the first embeddings and the second features of the near-end signal included in the second embeddings using an alignment block configured to use attention to align the first features and the second features;

obtaining attention weights output by the alignment block in response to aligning the first features and the second features;

concatenating the first embeddings, the second embeddings, and the attention weights into a first concatenated input using a first concatenation block;

performing acoustic echo cancellation (AEC) on the first concatenated input using a first group of Long Short-Term Memory (LSTM) blocks to remove the echo signal and to output third features,

concatenating the third features output by the first group of LSTM blocks and the attention weights output by the alignment block into a second concatenated input using a second concatenation block, and

performing personalized noise suppression (PNS) on the second concatenated input using a second group of LSTM blocks to remove speech from the one or more interfering speakers and to remove a residual echo output by the first group of LSTM blocks, and

outputting an audio signal comprising speech of the target speaker.

13 . The data processing system of claim 12 , wherein the first batch of training data includes speech of the target speaker, noise data, and echo data.

14 . The data processing system of claim 13 , wherein the second batch of training data includes the speech of the target speaker, the speech of the one or more interfering speakers who are different than the target speaker, and the noise data.

15 . The data processing system of claim 14 , wherein the second batch of training data further includes echo data.