Low latency audio packet loss concealment
The invention provides a method for real-time concealing errors in audio data packets. A Long Short-Term Memory (LSTM) neural network with a plurality of nodes is provided and pre-trained with audio data. A sequence of packets is received, each packet comprising a set of modified discrete cosine transform (MDCT) coefficients associated with a frame comprising time-domain samples of the audio signal. These MDCT coefficient data are applied to the LSTM neural network, and in case it is identified that a received packet is an erroneous packet, an output from the LSTM neural network is used to generate estimated MDCT co-efficients to provide a concealment packet to replace the erroneous packet. Preferably, the MDCT coefficients are normalized prior to applying to the LSTM neural network. This method can be performed in real-time. A low latency can be obtained and still with a high audio quality.
1 . A method for concealing errors, the method comprising:
training a plurality of different Long Short-Term Memory (LSTM) neural network configurations or LSTM neural network data sets using respective different specific audio inputs,
providing (P_L_NN) a LSTM neural network with a plurality of nodes, wherein the LSTM neural network has been pre-trained with audio data,
receiving (R_P) a sequence of packets each comprising a set of modified discrete cosine transform (MDCT) coefficients associated with a frame comprising time-domain samples of an audio signal,
classifying audio in response to the sequence of packets, and selecting one of the different LSTM neural network configurations or LSTM neural network data sets to be used for estimated MDCT coefficient generation accordingly,
applying (A_P_L_NN) the sequence of packets to the LSTM neural network,
identifying (I_E_P) in the sequence of packets a packet to be an erroneous packet,
generating (G_CFF) estimated MDCT coefficients to replace the set of MDCT coefficients of the erroneous packet in response to an output from the LSTM neural network,
generating (G_CP) a concealment packet based on the estimated MDCT coefficients, and
replacing (R_E_P) the erroneous packet with the concealment packet.
2 . The method according to claim 1 , comprising performing a normalizing procedure on the MDCT coefficients of each packet to arrive at a normalized set of MDCT coefficient for each packet, and applying the normalized set of MDCT coefficients to the LSTM neural network.
3 . The method according to claim 1 , wherein the LSTM neural network is pre-trained with audio data in a form of normalized sets of MDCT coefficients.
4 . The method according to claim 1 , comprising applying a post-processing of the estimated MDCT coefficients to modify the estimated MDCT coefficients prior to generating the concealment packet.
5 . The method according to claim 1 , wherein the sequence of packets are overlapping packets each representing audio samples, the sequence of packets optionally overlapping by 20-70%.
6 . The method according to claim 1 , wherein the LSTM neural network comprises a plurality of LSTM layers, wherein each LSTM layer comprises a plurality of LSTM nodes in parallel.
7 . The method according to claim 1 , wherein the LSTM neural network has one single LSTM layer of nodes, and optionally wherein outputs from nodes of the single LSTM layer are combined to provide the output of the LSTM neural network with a desired number of elements.
8 . The method according to claim 1 , comprising (a) training the LSTM neural network by a predetermined loss function using an audio input.
9 . The method according to claim 1 , wherein said classifying of audio comprises applying an acoustic scene classification algorithm on the sequence of packets to classify received audio with respect to input type-loading of pre-trained set of data for LSTM neural network.
10 . The method according to claim 1 , wherein transformation of audio samples into MDCT packets is performed prior to transmitting packets over a wireless transmission channel.
11 . The method according to claim 1 , wherein transformation of audio samples into MDCT packets is performed after receiving packets via a wireless transmission channel.
12 . The method of claim 1 , wherein the method is performed for one-way or two-way audio streaming.
13 . The method of claim 1 , further comprising:
training the LSTM neural network using a specific audio input selected from: a specific voice, a specific music instrument, or a specific music genre, or a specific person playing a specific instrument.
14 . A computer program product comprising a non-transitory computer-readable medium and with instructions that, when executed on a processor system, cause performance of operations comprising:
training a plurality of different Long Short-Term Memory (LSTM) neural network configurations or LSTM neural network data sets using respective different specific audio inputs,
providing (P_L_NN) a LSTM neural network with a plurality of nodes, wherein the LSTM neural network has been pre-trained with audio data,
receiving (R_P) a sequence of packets each comprising a set of modified discrete cosine transform (MDCT) coefficients associated with a frame comprising time-domain samples of an audio signal,
classifying audio in response to the sequence of packets, and selecting one of the different LSTM neural network configurations or LSTM neural network data sets to be used for estimated MDCT coefficient generation accordingly,
applying (A_P_L_NN) the sequence of packets to the LSTM neural network,
identifying (I_E_P) in the sequence of packets a packet to be an erroneous packet,
generating (G_CFF) estimated MDCT coefficients to replace the set of MDCT coefficients of the erroneous packet in response to an output from the LSTM neural network,
generating (G_CP) a concealment packet based on the estimated MDCT coefficients, and
replacing (R_E_P) the erroneous packet with the concealment packet.
15 . The non-transitory computer-readable medium of claim 14 , with additional instructions that, when executed on the processor system, cause performance of:
training the LSTM neural network using a specific audio input selected from: a specific voice, a specific music instrument, or a specific music genre, or a specific person playing a specific instrument.
16 . One or more devices, comprising:
a wireless receiver configured to receive a wireless signal representing a sequence of packets, and
a packet loss concealment system arranged to perform, on a processor system, to arrive at a modified sequence of packets comprising a concealment packet, operations comprising:
training a plurality of different Long Short-Term Memory (LSTM) neural network configurations or LSTM neural network data sets using respective different specific audio inputs,
providing (P_L_NN) a LSTM neural network with a plurality of nodes, wherein the LSTM neural network has been pre-trained with audio data,
receiving (R_P) the sequence of packets each comprising a set of modified discrete cosine transform (MDCT) coefficients associated with a frame comprising time-domain samples of an audio signal,
classifying audio in response to the sequence of packets, and selecting one of the different LSTM neural network configurations or LSTM neural network data sets to be used for estimated MDCT coefficient generation accordingly,
applying (A_P_L_NN) the sequence of packets to the LSTM neural network,
identifying (I_E_P) in the sequence of packets a packet to be an erroneous packet,
generating (G_CFF) estimated MDCT coefficients to replace the set of MDCT coefficients of the erroneous packet in response to an output from the LSTM neural network,
generating (G_CP) the concealment packet based on the estimated MDCT coefficients, and
replacing (R_E_P) the erroneous packet with the concealment packet.
17 . The one or more devices according to claim 16 , comprising an audio decoder arranged to receive the modified sequence of packets and to apply an MDCT based audio decoding algorithm so as to decode the modified sequence of packets into a sequence of decoded audio frames.
18 . The one or more devices according to claim 16 , the one or more devices being one of: a live performance base station, a wireless microphone, a wireless headset, a wireless intercom device, a teleconference system, a wireless audio monitor, and a virtual reality device.
19 . The one or more devices according to claim 16 , the one or more devices being arranged to generate an output audio signal in response to the received sequence of packets.
20 . The one or more devices according to claim 16 , wherein the sequence of packets represents samples of a time signal at a sample frequency of at least 8 kHz.
21 . The one or more devices according to claim 16 , further comprising an audio device comprising a wireless transmitter, wherein the audio device is arranged to generate the sequence of packets and to transmit the wireless signal by means of the wireless transmitter.
22 . The one or more devices according to claim 16 , wherein the packet loss concealment system is further arranged to perform:
training the LSTM neural network using a specific audio input selected from: a specific voice, a specific music instrument, or a specific music genre, or a specific person playing a specific instrument.