Method for enhancing quality of audio data, and device using the same
Provided is a method of enhancing quality of audio data which comprise obtaining a spectrum of mixed audio data including noise, inputting two-dimensional (2D) input data corresponding to the spectrum to a convolutional network including a downsampling process and an upsampling process to obtain output data of the convolutional network, generating a mask for removing noise included in the audio data based on the obtained output data and removing noise from the mixed audio data using the generated mask, wherein, in the convolutional network, the downsampling process and the upsampling process are performed on a first axis of the 2D input data, and remaining processes other than the downsampling process and the upsampling process are performed on the first axis and a second axis.
1. A method of enhancing quality of audio data, the method comprising:
obtaining a spectrum of mixed audio data including noise;
inputting two-dimensional (2D) input data corresponding to the spectrum to a convolutional network including a downsampling process and an upsampling process to obtain output data of the convolutional network;
generating a mask for removing noise included in the audio data based on the obtained output data; and
removing noise from the mixed audio data using the generated mask,
wherein, in the convolutional network which is a U-NET convolutional network, the downsampling process and the upsampling process are performed only on a frequency axis of the 2D input data, and remaining processes other than the downsampling process and the upsampling process are performed on the frequency axis and a time axis, and
wherein the method further comprises:
performing a causal convolution on the 2D input data on the time axis,
wherein the performing of the causal convolution comprises:
performing zero padding on data of a preset size corresponding to the past relative to the time axis in the 2D input data.
2. The method of claim 1 , wherein the performing of the causal convolution is performed on the time axis.
3. The method of claim 1 , wherein a batch normalization process is performed before the downsampling process.
4. The method of claim 1 , wherein the obtaining of the spectrum of mixed audio data including noise comprises:
obtaining the spectrum by applying a short-time Fourier transform (STFT) to the mixed audio data including noise.
5. The method of claim 1 , the method being performed on the audio data collected in real time.
6. An audio data processing device comprising:
an audio data pre-processor configured to obtain a spectrum of mixed audio data including noise;
an encoder and a decoder configured to input 2D input data corresponding to the spectrum to a convolutional network including a downsampling process and an upsampling process to obtain output data of the convolutional network; and
an audio data post-processor configured to generate a mask for removing noise included in the audio data based on the obtained output data, and to remove noise from the mixed audio data using the generated mask,
wherein, in the convolutional network which is a U-NET convolutional network, the downsampling process and the upsampling process are performed only on a frequency axis of the 2D input data, and remaining processes other than the downsampling process and the upsampling process are performed on the frequency axis and a time axis, and
wherein the encoder and the decoder performs a causal convolution on the 2D input data on the time axis, and
wherein the causal convolution performs zero padding on data of a preset size corresponding to the past relative to the time axis in the 2D input data.