Method and apparatus for designing and testing audio codec by using white noise modeling
Provided is a method and apparatus for designing and testing an audio codec using quantization based on white noise modeling. A neural network-based audio encoder design method includes generating a quantized latent vector and a reconstructed signal corresponding to an input signal by using a white noise modeling-based quantization process, computing a total loss for training a neural network-based audio codec, based on the input signal, the reconstruction signal, and the quantized latent vector, training the neural network-based audio codec by using the total loss, and validating the trained neural network-based audio codec to select the best neural network-based audio codec.
1. A method of designing a neural network-based audio codec, the method comprising:
generating a quantized latent vector and a reconstructed signal corresponding to an input signal by using a white noise modeling-based quantization process;
computing a total loss for training of the neural network-based audio codec, based on the input signal, the reconstructed signal, and the quantized latent vector;
training the neural network-based audio codec by using the total loss; and
validating the trained neural network-based audio codec to select the best neural network-based audio codec,
wherein the computing of the total loss comprises:
calculating a reconstruction loss term as mean squared error (MSE) between the input signal and the reconstructed signal, a bit-rate control loss term as an entropy of the quantized latent vector, and a perceptual loss term reflecting human perceptual characteristics, respectively; and
calculating the total loss by adding the reconstruction loss term, the bit-rate control loss term, and the perceptual loss term,
wherein the reconstruction loss term is determined based on a square of an L2-norm of a difference between the input signal and the reconstructed signal,
wherein the bit-rate control loss term is determined based on probability distribution for a latent vector with added random noise,
wherein the latent vector with added random noise is generated by adding a random noise to a latent vector output from an encoder of the neural network-based audio codec that receives the input signal, and
wherein the quantized latent vector is generated by de-warping the latent vector with added random noise.
2. The method of claim 1 , wherein the generating of the reconstructed signal and the quantized latent vector comprises:
feeding the input signal into the encoder of the neural network-based audio codec;
quantizing the latent vector resulting from the encoder by using white noise modeling; and
generating the reconstructed signal for the input signal by feeding the quantized latent vector into a decoder of the neural network-based audio codec.
3. The method of claim 2 , wherein the quantizing of the latent vector comprises:
warping the latent vector to constrain each element in the latent vector within a predetermined range;
adding a random noise vector corresponding to quantization noise to the warped latent vector to generate a noisy latent vector; and
de-warping the noisy latent vector to output the quantized latent vector.
4. The method of claim 3 , wherein both distribution and range of the generated random noise are determined according to a quantization step of the neural network-based audio codec.
5. The method of claim 1 , wherein the training of the neural network comprises performing training of the neural network-based audio codec by backpropagating the total loss through the neural network-based audio codec.
6. The method of claim 1 , wherein the validating of the neural network-based audio codec comprises evaluating the trained neural network-based audio codec in terms of validation score as the total loss or objective quality metrics by feeding a signal of a validation database (DB) into the trained neural network-based audio codec, comparing the evaluated validation score with the best validation score, and selecting the trained neural network-based audio codec as the best neural network-based audio codec and updating the best validation score with the evaluated validation score if the evaluated validation score is better than the best validation score.
7. A neural network-based audio codec design apparatus comprising:
an audio codec training device configured to generate a quantized latent vector and a reconstructed signal corresponding to an input signal by using a white noise modeling-based quantization process, compute a total loss for training of a neural network-based audio codec, based on the input signal, the reconstructed signal, and the quantized latent vector, and train the neural network-based audio codec by using the total loss; and
an audio codec validation device configured to validate the trained neural network-based audio codec to select the best neural network-based audio codec,
wherein the audio codec training device is configured to calculate a reconstruction loss term as mean squared error (MSE) between the input signal and the reconstructed signal, a bit-rate control loss term as an entropy of the quantized latent vector, and a perceptual loss term reflecting human perceptual characteristics, and calculate the total loss by adding the reconstruction loss term, the bit-rate control loss term, and the perceptual loss term,
wherein the reconstruction loss term is determined based on a square of an L2-norm of a difference between the input signal and the reconstructed signal,
wherein the bit-rate control loss term is determined based on probability distribution for a latent vector with added random noise,
wherein the latent vector with added random noise is generated by adding a random noise to a latent vector output from an encoder of the neural network-based audio codec that receives the input signal, and
wherein the quantized latent vector is generated by de-warping the latent vector with added random noise.
8. The neural network-based audio codec design apparatus of claim 7 , wherein the audio codec training device is configured to feed the input signal into the encoder of the neural network-based audio codec, quantize the latent vector resulting from the encoder by using white noise modeling, and generate the reconstructed signal for the input signal by feeding the quantized latent vector into a decoder of the neural network-based audio codec.
9. The neural network-based audio codec design apparatus of claim 8 , wherein the audio codec training device is configured to warp the latent vector to constrain each element in the latent vector within a predetermined range, add a random noise vector corresponding to quantization noise to the warped latent vector to generate a noisy latent vector, and de-warp the noisy latent vector to output the quantized latent vector.
10. The neural network-based audio codec design apparatus of claim 9 , wherein both distribution and range of the generated random noise are determined according to a quantization step of the neural network-based audio codec.
11. The neural network-based audio codec design apparatus of claim 7 , wherein the audio codec training device is configured to perform training of the neural network-based audio codec by backpropagating the total loss through the neural network-based audio codec.
12. The neural network-based audio codec design apparatus of claim 7 , wherein the audio codec validation device is configured to evaluate the trained neural network-based audio codec in terms of validation score as the total loss or objective quality metrics by feeding a signal of a validation database (DB) into the trained neural network-based audio codec, compare the evaluated validation score with the best validation score, and select the trained neural network-based audio codec as the best neural network-based audio codec and update the best validation score with the evaluated validation score if the evaluated validation score is better than the best validation score.