Embedded enablement for optimal gain-mask machine learning, audio channel-based signal enhancement

1 . A system comprising:

memory storing instructions; and

a processing device coupled to the memory, the processing device to execute the instructions to:

receive a noisy audio signal from an audio receiver;

pass the noisy audio signal through a deep neural network (DNN) model to generate a mask of a magnitude spectrogram of the noisy audio signal;

retrieve a clean-only audio spectra and a noise-only spectra of one or more frequencies that exist within the noisy audio signal;

generate a loss function as a combination of the clean-only audio spectra multiplied by the mask, which comprises a filtered audio spectrum, and the noise-only spectra multiplied by the mask, which comprises a filtered noise spectrum, and wherein the loss function comprises:

a scalar value multiplied by the filtered noise spectrum; and

one minus the scalar value multiplied by a difference between the filtered audio spectrum and the clean-only audio spectra; and

train the DNN model while minimizing the loss function to generate a trained DNN model useable in audio noise suppression and dereverberation.

2 . The system of claim 1 , wherein the processing device is further to:

perform a signal-to-noise ratio (SNR) estimation and normalization on the clean-only audio spectra to generate normalized estimated SNR values;

measure noise spectral stationarity of the noise-only spectra to generate a measured noise spectral stationarity values that characterize the flatness of the noise-only spectra; and

incorporate weighted versions of the normalized estimated SNR values and the measured noise spectral stationarity values in a regression function that generates a tradeoff parameter output useable in training the DNN model.

3 . The system of claim 1 , wherein the processing device is further to perform, using the trained DNN model, at least one of channel-based noise suppression or channel-based dereverberation during active operation.

4 . The system of claim 3 , wherein the processing device is further to:

process the clean-only audio spectra with a Short-Term Fourier Transform (STFT) function and determine a first magnitude of the processed clean-only audio spectra;

process the noisy audio signal with the STFT function;

determine a second magnitude and a first phase of the processed noisy audio signal;

input, to the trained DNN model, the first magnitude as a target audio signal and the second magnitude as a predicted audio signal;

employ an output of the trained DNN model as an output magnitude and the phase as an output phase to generate an output audio signal; and

apply an inverse STFT function to the output audio signal to generate a denoised audio signal.

5 . The system of claim 1 , wherein the mask is an ideal ratio mask, comprising only ones and zeros, determined a-priori based on at least the clean-only audio spectra.

6 . The system of claim 1 , wherein the DNN model comprises:

multiple sets of intermediate layers, each set including a convolutional layer followed by a batch normalization layer followed by a sigmoid layer; and

a final regression layer to learn from outputs of the multiple sets of intermediate layers and estimate values of the mask.

7 . The system of claim 1 , wherein the scalar value is an aggressiveness parameter that is tunable to balance a tradeoff between removing the noise-only spectra and distorting the filtered audio spectrum.

8 . The system of claim 1 , wherein the scalar value is between a zero value and a one value, and wherein the processing device is further to tune the scalar value such as to minimize both the filtered noise spectrum and distortion of the filtered audio spectrum.

9 . The system of claim 1 , wherein the processing device is further to:

determine an ideal ratio mask as the clean-only audio spectra divided by a combination of the clean-only audio spectra and the noise-only spectra;

determine a time-frequency-based loss function as an absolute magnitude of a difference of a predicted ratio mask and the ideal ratio mask over time and the one or more frequencies; and

modify the loss function with the time-frequency-based loss function.

10 . A computer-implemented method, performed by a processing device, comprising:

receiving a noisy audio signal from an audio receiver;

passing the noisy audio signal through a deep neural network (DNN) model to generate a mask of a magnitude spectrogram of the noisy audio signal;

retrieving a clean-only audio spectra and a noise-only spectra of one or more frequencies that exist within the noisy audio signal;

generating a loss function as a combination of the clean-only audio spectra multiplied by the mask, which comprises a filtered audio spectrum, and the noise-only spectra multiplied by the mask, which comprises a filtered noise spectrum, and wherein the loss function comprises:

a scalar value multiplied by the filtered noise spectrum; and

one minus the scalar value multiplied by a difference between the filtered audio spectrum and the clean-only audio spectra; and

training the DNN model while minimizing the loss function to generate a trained DNN model useable in audio noise suppression and dereverberation.

11 . The computer-implemented method of claim 10 , further comprising:

perform a signal-to-noise ratio (SNR) estimation and normalization on the clean-only audio spectra to generate normalized estimated SNR values;

measure noise spectral stationarity of the noise-only spectra to generate a measured noise spectral stationarity values that characterize the flatness of the noise-only spectra; and

incorporate weighted versions of the normalized estimated SNR values and the measured noise spectral stationarity values in a regression function that generates a tradeoff parameter output useable in training the DNN model.

12 . The computer-implemented method of claim 10 , further comprising performing, using the trained DNN model, at least one of channel-based noise suppression or channel-based dereverberation during active operation.

13 . The computer-implemented method of claim 12 , further comprising:

processing the clean-only audio spectra with a Short-Term Fourier Transform (STFT) function and determine a first magnitude of the processed clean-only audio spectra;

processing the noisy audio signal with the STFT function;

determine a second magnitude and a first phase of the processed noisy audio signal;

inputting, to the trained DNN model, the first magnitude as a target audio signal and the second magnitude as a predicted audio signal;

employing an output of the trained DNN model as an output magnitude and the phase as an output phase to generate an output audio signal; and

applying an inverse STFT function to the output audio signal to generate a denoised audio signal.

14 . The computer-implemented method of claim 10 , wherein the mask is an ideal mask, comprising only ones and zeros, determined a-priori based on at least the clean-only audio spectra.

15 . The computer-implemented method of claim 10 , wherein the DNN model comprises:

multiple sets of intermediate layers, each set including a convolutional layer followed by a batch normalization layer followed by a sigmoid layer; and

a final regression layer to learn from outputs of the multiple sets of intermediate layers and estimate values of the mask.

16 . The computer-implemented method of claim 10 , further comprising tuning the scalar value as an aggressiveness parameter to balance a tradeoff between removing the noise-only spectra and distorting the filtered audio spectrum.

17 . The computer-implemented method of claim 10 , wherein the scalar value is between a zero value and a one value, the method further comprising tuning the scalar value such as to minimize both the filtered noise spectrum and distortion of the filtered audio spectrum.

18 . The computer-implemented method of claim 10 , further comprising:

determining an ideal ratio mask as the clean-only audio spectra divided by a combination of the clean-only audio spectra and the noise-only spectra;

determining a time-frequency-based loss function as an absolute magnitude of a difference of a predicted ratio mask and the ideal ratio mask over time and the one or more frequencies; and

modifying the loss function with the time-frequency-based loss function.

19 . A non-transitory computer-readable storage medium storing instructions, which when executed, cause a processing device to perform operations comprising:

receiving a noisy audio signal from an audio receiver;

passing the noisy audio signal through a deep neural network (DNN) model to generate a mask of a magnitude spectrogram of the noisy audio signal;

retrieving a clean-only audio spectra and a noise-only spectra of one or more frequencies that exist within the noisy audio signal;

generating a loss function as a combination of the clean-only audio spectra multiplied by the mask, which comprises a filtered audio spectrum, and the noise-only spectra multiplied by the mask, which comprises a filtered noise spectrum, and wherein the loss function comprises:

a scalar value multiplied by the filtered noise spectrum; and

one minus the scalar value multiplied by a difference between the filtered audio spectrum and the clean-only audio spectra; and

training the DNN model while minimizing the loss function to generate a trained DNN model useable in audio noise suppression and dereverberation.

20 . The non-transitory computer-readable storage medium of claim 19 , wherein the operations further comprise:

perform a signal-to-noise ratio (SNR) estimation and normalization on the clean-only audio spectra to generate normalized estimated SNR values;

measure noise spectral stationarity of the noise-only spectra to generate a measured noise spectral stationarity values that characterize the flatness of the noise-only spectra; and

incorporate weighted versions of the normalized estimated SNR values and the measured noise spectral stationarity values in a regression function that generates a tradeoff parameter output useable in training the DNN model.

21 . The non-transitory computer-readable storage medium of claim 19 , wherein the operations further comprise performing, using the trained DNN model, at least one of channel-based noise suppression or channel-based dereverberation during active operation.

22 . The non-transitory computer-readable storage medium of claim 21 , wherein the operations further comprise:

processing the clean-only audio spectra with a Short-Term Fourier Transform (STFT) function and determine a first magnitude of the processed clean-only audio spectra;

processing the noisy audio signal with the STFT function;

determine a second magnitude and a first phase of the processed noisy audio signal;

inputting, to the trained DNN model, the first magnitude as a target audio signal and the second magnitude as a predicted audio signal;

employing an output of the trained DNN model as an output magnitude and the phase as an output phase to generate an output audio signal; and

applying an inverse STFT function to the output audio signal to generate a denoised audio signal.

23 . The non-transitory computer-readable storage medium of claim 19 , wherein the operations further comprise tuning the scalar value as an aggressiveness parameter to balance a tradeoff between removing the noise-only spectra and distorting the filtered audio spectrum.

24 . The non-transitory computer-readable storage medium of claim 19 , wherein the scalar value is between a zero value and a one value, wherein the operations further comprise tuning the scalar value such as to minimize both the filtered noise spectrum and distortion of the filtered audio spectrum.

25 . The non-transitory computer-readable storage medium of claim 19 , wherein the operations further comprise:

determining an ideal ratio mask as the clean-only audio spectra divided by a combination of the clean-only audio spectra and the noise-only spectra;

determining a time-frequency-based loss function as an absolute magnitude of a difference of a predicted ratio mask and the ideal ratio mask over time and the one or more frequencies; and

modifying the loss function with the time-frequency-based loss function.