Methods and apparatuses for noise reduction based on time and frequency analysis using deep learning

1. A noise cancellation method using a computer system, the noise cancellation method comprising:

generating a first voice signal by canceling a first portion of noise included in an input voice signal using a first network, the first network being a trained u-net structure, and the first portion of the noise being in a time domain;

applying a first window to the first voice signal to obtain a first windowed voice signal;

performing a fast Fourier transform (FFT) on the first windowed voice signal to acquire a magnitude signal and a phase signal;

acquiring a mask using a second network based on the magnitude signal, the second network being another trained u-net structure;

applying the mask to the magnitude signal to obtain a masked magnitude signal;

generating a second voice signal by canceling a second portion of the noise by performing an inverse fast Fourier transform (IFFT) on the first windowed voice signal based on the masked magnitude signal and the phase signal; and

applying a second window to the second voice signal to obtain a second windowed voice signal.

2. The noise cancellation method of claim 1 , wherein at least one of the first window or the second window comprises a Kaiser-Bessel-derived window for time domain aliasing cancellation (TDAC) in modified discrete cosine transform (MDCT).

3. The noise cancellation method of claim 1 , wherein

the magnitude signal includes a first magnitude signal and a second magnitude signal; and

the acquiring the mask comprises:

acquiring a first mask using the second network based on the first magnitude signal, the first magnitude signal being in a first frequency band,

dividing the second magnitude signal into a plurality of second magnitude sub-signals according to bandwidth, the second magnitude signal being in a second frequency band greater than the first frequency band,

calculating an average energy for each of the plurality of second magnitude sub-signals, and

acquiring a second mask using the second network based on the average energy for each of the plurality of second magnitude sub-signals.

4. The noise cancellation method of claim 3 , wherein the dividing divides the second magnitude signal into the plurality of second magnitude sub-signals by dividing the second frequency band based on a bark scale unit.

5. The noise cancellation method of claim 3 , wherein

the first mask is an ideal ratio mask (IRM) for the first magnitude signal and the second mask is an IRM for the average energy; and

the applying the mask to the magnitude signal comprises multiplying the first mask by the first magnitude signal and multiplying the second mask by the second magnitude signal.

6. The noise cancellation method of claim 3 , wherein the acquiring the first mask comprises:

calculating a number of Mel-frequency cepstral coefficients (MFCCs) based on the first magnitude signal; and

acquiring the first mask using the second network based on the MFCCs.

7. The noise cancellation method of claim 3 , wherein the acquiring the first mask comprises:

calculating a zero-crossing rate (ZCR) based on the first magnitude signal; and

acquiring the first mask using the second network based on the ZCR.

8. The noise cancellation method of claim 1 , wherein the generating the second voice signal comprises:

estimating a denoised magnitude signal by multiplying the magnitude signal and the mask;

recovering an FFT coefficient based on the denoised magnitude signal and the phase signal; and

recovering the second voice signal by performing the IFFT based on the FFT coefficient.

9. The noise cancellation method of claim 1 , wherein the input voice signal comprises a plurality of frames.

10. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause a computer system including the at least one processor to perform the noise cancellation method of claim 1 .

11. A computer system for cancelling noise, the computer system comprising:

a memory storing computer-readable instructions; and

at least one processor configured to execute the computer-readable instructions to cause the computer systems to,

generate a first voice signal by canceling a first portion of noise included in an input voice signal using a first network, the first network being a trained u-net structure, and the first portion of the noise being in a time domain,

apply a first window to the first voice signal to obtain a first windowed voice signal,

perform a fast Fourier transform (FFT) on the first windowed voice signal to acquire a magnitude signal and a phase signal,

acquire a mask using a second network based on the magnitude signal, the second network being another trained u-net structure,

apply the mask to the magnitude signal to obtain a masked magnitude signal,

generate a second voice signal by canceling a second portion of the noise by performing an inverse fast Fourier transform (IFFT) on the first windowed voice signal based on the masked magnitude signal and the phase signal, and

apply a second window to the second voice signal to obtain a second windowed voice signal.

12. The computer system of claim 11 , wherein at least one of the first window or the second window comprises a Kaiser-Bessel-derived window for time domain aliasing cancellation (TDAC) in modified discrete cosine transform (MDCT).

13. The computer system of claim 11 , wherein

the magnitude signal includes a first magnitude signal and a second magnitude signal; and

the at least one processor is configured to cause the computer system to,

acquire a first mask using the second network based on the first magnitude signal, the first magnitude signal being in a first frequency band,

divide the second magnitude signal into a plurality of second magnitude sub-signals according to bandwidth, the second magnitude signal being in a second frequency band greater than the first frequency band,

calculate an average energy for each of the plurality of second magnitude sub-signals, and

acquire a second mask using the second network based on the average energy for each of the plurality of second magnitude sub-signals.

14. The computer system of claim 11 , wherein the at least one processor is configured to cause the computer system to:

estimate a denoised magnitude signal by multiplying the magnitude signal and the mask;

recover an FFT coefficient based on the denoised magnitude signal and the phase signal; and

recover the second voice signal by performing the IFFT based on the FFT coefficient.

15. The computer system of claim 13 , wherein the at least one processor is configured to cause the computer system to divide the second magnitude signal into the plurality of second magnitude sub-signals by dividing the second frequency band based on a bark scale unit.

16. The computer system of claim 13 , wherein the at least one processor is configured to cause the computer system to apply the mask to the magnitude signal by multiplying the first mask by the first magnitude signal and multiplying the second mask by the second magnitude signal.

17. The noise cancellation method of claim 1 , further comprising:

generating an audio signal based on the second windowed voice signal; and

driving a speaker to output the audio signal.

18. The noise cancellation method of claim 1 , wherein the second portion of the noise is in a frequency domain.

19. The computer system of claim 11 , wherein the at least one processor is configured to cause the computer system to:

generate an audio signal based on the second windowed voice signal; and

drive a speaker to output the audio signal.

20. The computer system of claim 11 , wherein the second portion of the noise is in a frequency domain.