Real-time packet loss concealment using deep generative networks
The present disclosure relates to a method and system for performing packet loss concealment using a neural network system. The method comprises obtaining a representation of an incomplete audio signal, inputting the representation of the incomplete audio signal to an encoder neural network and outputting a latent representation of a predicted complete audio signal. The latent representation is input to a decoder neural network which outputs a representation of a predicted complete audio signal comprising a reconstruction of the original portion of the complete audio signal, wherein said encoder neural network and said decoder neural network have been trained with an adversarial neural network.
1. A method for packet loss concealment of an incomplete audio signal, the incomplete audio signal comprising a substitute signal portion replacing an original signal portion of a complete audio signal, the method comprising:
obtaining a representation of the incomplete audio signal;
inputting the representation of the incomplete audio signal to an encoder neural network trained to predict a latent representation of a complete audio signal given a representation of an incomplete audio signal;
outputting, by the encoder neural network, a latent representation of a predicted complete audio signal;
quantizing the latent representation of the complete audio signal to obtain a quantized latent representation, wherein the quantized latent representation is formed by selecting a set of tokens out of a predetermined vocabulary set of tokens;
conditioning, with at least one token of the quantized latent representation, a generative neural network, wherein the generative neural network is trained to predict a token of the set of tokens provided at least one different token of the set of tokens;
outputting by the generative neural network a predicted token of the latent representation and a confidence metric associated with the predicted token;
based on the confidence metric of the predicted token, replacing a corresponding token of the quantized latent representation with the predicted token,
inputting the quantized latent representation of the predicted complete audio signal to a decoder neural network trained to predict a representation of a complete audio signal given a latent representation of a complete audio signal; and
outputting, by the decoder neural network, a representation of the predicted complete audio signal comprising a reconstruction of the original portion of the complete audio signal, wherein said encoder neural network and said decoder neural network have been trained with an adversarial neural network.
2. The method according to claim 1 , wherein the quantized latent representation is a first quantized latent representation, the method further comprising:
quantizing the latent representation of the complete audio signal to obtain a second quantized latent representation, wherein the second quantized latent representation is formed by selecting a second set of tokens out of a second predetermined vocabulary set of tokens; and
inputting the first and second quantized latent representation to the decoder neural network.
3. The method according to claim 2 , wherein each token is represented by a quantization vector and wherein inputting the first and second quantized latent representation to the decoder neural network comprises:
concatenating or adding the quantization vectors of the first quantized latent representation with corresponding quantization vectors of the second quantized latent representation.
4. The method according to claim 2 , further comprising:
providing the second quantized latent representation to a second generative neural network, wherein the second generative neural network is trained to predict a token of the second set of tokens given at least one different token of the second set of tokens;
outputting by the second generative neural network a predicted token of the second latent representation and a confidence metric associated with the predicted token; and
based on the confidence metric of the predicted token of the second latent representation, replacing a corresponding token of the second quantized latent representation with the predicted token of the second generative neural network.
5. The method according to claim 1 , further comprising:
obtaining at least one random variable, and
inputting the at least one random variable to the decoder neural network, wherein the decoder neural network is trained to predict a representation of the complete audio signal given the latent representation of a complete audio signal and the at least one random variable.
6. The method according to claim 1 , further comprising:
transforming the incomplete audio signal into the representation of the incomplete audio signal.
7. The method according to claim 1 , wherein outputting, by the decoder neural network, a representation of the predicted complete audio signal comprises outputting a time domain representation of the predicted complete audio signal.
8. The method according to claim 1 , further comprising:
inverse transforming the representation of the predicted complete audio signal into a time domain predicted complete audio signal.
9. The method according to claim 1 , wherein the representation is one of a frequency representation, a time representation, a filter bank representation and/or a feature domain representation.
10. The method according to claim 9 , wherein the representation is Mel-frequency cepstral coefficients or a short-time Fourier transform.
11. The method according to claim 1 , wherein the decoder neural network comprises at least one causal convolutional layer and at least one subsequent residual layer, the method further comprising:
performing causal up-sampling by the causal convolutional layer to form an up-sampled representation; and
transforming the up-sampled representation by the at least one residual layer.
12. The method according to claim 2 , further comprising:
determining a window function based on the reconstructed original portion of the predicted complete audio signal and a directly subsequent portion of the audio signal; and
applying the determined window function on the reconstructed original portion of the predicted complete audio.
13. The method according to claim 12 , wherein determining a window function comprises:
inputting a representation of the reconstructed original portion of the predicted complete audio signal and the directly subsequent portion of the predicted complete audio signal to a window function neural network trained to predict a window function given a reconstructed audio signal portion and the subsequent portion of the audio signal.
14. A computer implemented neural network system for packet loss concealment of an audio signal, wherein the audio signal comprises a substitute signal portion replacing an original signal portion of a complete audio signal, said system comprising:
an input unit, configured to obtain a representation of the incomplete audio signal,
an encoder neural network trained to predict a latent representation of a complete audio signal given a representation of an incomplete audio signal and configured to receive the representation of the incomplete audio signal and output a latent representation of a predicted complete audio signal,
a quantization block, configured to quantize the latent representation of the complete audio signal to obtain a quantized latent representation, wherein the quantized latent representation is formed by selecting a set of tokens out of a predetermined vocabulary set of tokens,
a generative neural network, wherein the generative neural network being trained to predict a token of the set of tokens provided at least one different token of the set of tokens, the generative neural network being configured to be conditioned with at least one token of the quantized latent representation and outputting a predicted token of the latent representation and a confidence metric associated with the predicted token,
wherein, based on the confidence metric of the predicted token, replacing a corresponding token of the quantized latent representation with the predicted token,
a decoder neural network trained to predict a representation of a complete audio signal given a quantized latent representation of a complete audio signal, and configured to receive the latent representation of the predicted complete audio signal and output a representation of a complete audio signal,
an output unit configured to output a representation of the predicted complete audio signal comprising a reconstruction of the original portion of the complete audio signal, wherein said encoder neural network and said decoder neural network have been trained with an adversarial neural network.
15. A method for training a neural network system for packet loss concealment, the method comprising:
obtaining the neural network system for packet loss concealment of claim 11 ;
obtaining a discriminator neural network;
obtaining a set of training data; and
training the neural network system in conjunction with the discriminator neural network using the set of training data in generative-adversarial training mode by providing the set of training data to the neural network system and providing an output of the neural network system to the discriminator neural network.
16. The method for training a neural network system for packet loss concealment according to claim 15 , wherein the training data comprises a random noise signal.
17. The method for training a neural network system for packet loss concealment according to claim 15 , wherein the training data comprises at least one example of a recorded audio signal.
18. The method for training a neural network system for packet loss concealment according to claim 15 , wherein the discriminator neural network comprises two neural network branches, the method further comprising:
providing an output of the neural network system to a first neural network branch of said two neural network branches, said output comprising a representation of an audio signal with a reconstructed audio signal portion,
outputting by the first neural network branch a likelihood measure that the output of the neural network system comprises a representation of a reconstructed audio signal portion;
down-sampling the output of the neural network system;
providing the down-sampled output of the neural network system to a second neural network branch of said two neural network branches;
outputting by the second neural network branch a second likelihood measure that the down-sampled output of the neural network system comprises a representation of a reconstructed audio signal portion;
determining an aggregate likelihood measure based on the likelihood measure and the second likelihood measure; and
using the aggregate likelihood measure in the generative-adversarial training.
19. The method for training a neural network system for packet loss concealment according to claim 15 , wherein the discriminator network outputs a likelihood measure indicating the likelihood that the output of the neural network system comprises a complete audio signal and wherein training the neural network system in conjunction with the discriminator neural network comprises:
performing training in a first and second mode of operation, wherein
the first mode of operation comprises:
inputting the set of training data to the neural network system,
changing the internal weights of the discriminator neural network to decrease the likelihood measure,
maintaining the internal weights of the neural network system,
the second mode of operation comprises:
inputting the set of training data to the neural network system,
changing the internal weights of the neural network system to increase the likelihood measure,
maintaining the internal weights of the discriminator neural network.
20. The method for training a neural network system for packet loss concealment according to claim 19 , further comprising
providing a second set of training data, wherein the second set of training data comprises at least one example of audio data,
performing training in a third mode of operation, wherein
the third mode of operation comprises:
inputting the second set of training data to the discriminator neural network, and
changing the internal weights of the discriminator neural network to increase the likelihood measure.