Audio packet loss compensation method and apparatus and electronic device

1 . A method for audio packet loss compensation processing, comprising:

acquiring time domain signals of a first number of existing audio frames in a case that an audio packet loss occurs;

carrying out acoustic feature extraction on the time domain signals of the first number of the existing audio frames to generate acoustic features of the existing audio frames;

carrying out, by a one-dimensional convolutional network, semantic feature extraction on the time domain signals of the first number of the existing audio frames, and carrying out, by a GRU network, semantic understanding based on the extracted semantic features to generate semantic expressions at a phoneme level corresponding to the existing audio frames;

predicting a time domain signal of a current audio frame according to the acoustic features and the semantic expressions, for audio packet loss compensation; and

performing audio packet loss compensation using the predicted time domain signal.

2 . The method of claim 1 , wherein: the carrying out the acoustic feature extraction on the time domain signals of the first number of the existing audio frames to generate the acoustic features of the existing audio frames comprises:

carrying out, by the one-dimensional convolutional network with multiple branches, acoustic feature extraction on the time domain signals of the first number of the existing audio frames to generate multiple low-dimensional acoustic features; and

carrying out aggregation processing on the multiple low-dimensional acoustic features to generate the acoustic features of the existing audio frames.

3 . The method of claim 2 , wherein: the carrying out aggregation processing on the multiple low-dimensional acoustic features to generate acoustic features of the existing audio frames comprises:

aggregating, by an attention module, the multiple low-dimensional acoustic features to generate the acoustic features of the existing audio frames.

4 . The method of claim 2 , further comprising:

buffering acoustic features of a preset number of the existing audio frames, for prediction processing of subsequent audio frames.

5 . The method of claim 1 , wherein: the predicting the time domain signal of the current audio frame according to the acoustic features and the semantic expressions comprises:

predicting, by the GRU network, the time domain signal of the current audio frame according to the acoustic features and the semantic expressions.

6 . The method of claim 5 , wherein the GRU network uses sparse model parameters.

7 . An apparatus for audio packet loss compensation processing, comprising:

a memory configured to store a program; and

one or more processors configured to run the program stored in the memory, to execute operations comprising:

acquiring a time domain signal and a frame state of a current audio frame;

in response to the frame state being a lost state:

predicting the time domain signal of the current audio frame by using an audio prediction model according to time domain signals of a first number of existing audio frames before the current audio frame by:

generating acoustic features of the existing audio frames;

carrying out, by a one-dimensional convolutional network, semantic feature extraction on the time domain signals of the first number of the existing audio frames, and carrying out, by a GRU network, semantic understanding based on the extracted semantic features to generate semantic expressions at a phoneme level corresponding to the existing audio frames;

predicting the time domain signal of the current audio frame according to the acoustic features and the semantic expressions; and

outputting a prediction result as the current audio frame, and performing audio packet loss compensation using the predicted time domain signal; or

in response to the frame state is an intact state:

training the audio prediction model by using the time domain signal of the current audio frame and the time domain signals of the first number of the existing audio frames before the current audio frame as training data;

updating the audio prediction model; and

outputting the current audio frame in the intact state.

8 . The apparatus method of claim 7 , wherein: the predicting the time domain signal of the current audio frame according to the time domain signals of the first number of the existing audio frames before the current audio frame further comprises:

carrying out acoustic feature extraction on the time domain signals of the first number of the existing audio frames to generate the acoustic features of the existing audio frames.

9 . The apparatus method of claim 7 , wherein: the carrying out the acoustic feature extraction on the time domain signals of the first number of the existing audio frames to generate the acoustic features of the existing audio frames further comprises:

carrying out, by the one-dimensional convolutional network with multiple branches, the acoustic feature extraction on the time domain signals of the first number of the existing audio frames to generate multiple low-dimensional acoustic features; and

carrying out aggregation processing on the multiple low-dimensional acoustic features to generate the acoustic features of the existing audio frames.

10 . The apparatus method of claim 7 , wherein: the predicting the time domain signal of the current audio frame according to the acoustic features and the semantic expressions comprises:

predicting, by the GRU network, the time domain signal of the current audio frame according to the acoustic features and the semantic expressions.

11 . The apparatus method of claim 7 , wherein: the training the audio prediction model by using the time domain signal of the current audio frame and the time domain signals of the first number of the existing audio frames before the current audio frame as the training data comprises:

adjusting model parameters of the audio prediction model according to a difference between the current audio frame and an output result of a training model by taking the time domain signals of the first number of the existing audio frames that are consecutive with the current audio frame as an input of the training model.

12 . The apparatus method of claim 7 , wherein: the training the audio prediction model by using the time domain signal of the current audio frame and the time domain signals of the first number of the existing audio frames before the current audio frame as the training data comprises:

carrying out a prediction training for consecutive frame loss, with a preset probability, by using the time domain signals of the first number of the existing audio frames with an interval from the current audio frame as an input of a training model; and

adjusting model parameters of the audio prediction model according to a difference between the current audio frame and an output result of the training model.

13 . The apparatus method of claim 7 , wherein: the audio prediction model is configured in correspondence to a live streaming terminal, and the method further comprises:

carrying out fusion processing on multiple audio prediction models at a preset time interval; and

configuring the fused audio prediction models in correspondence to each live streaming terminal.

14 . The apparatus method of claim 13 , wherein the fusion processing comprises:

carrying out weighted fusion processing on each live streaming terminal, wherein a weight value of the audio prediction model at a targeted live streaming terminal is greater than weight values of the audio prediction models at other live streaming terminals.

15 . The apparatus method of claim 7 , wherein the audio prediction model is a model trained offline by using a comprehensive training data set.

16 . An electronic device, comprising:

a memory configured to store a program; and

one or more processors configured to run the program stored in the memory to cause the electronic device to perform operations comprising:

acquiring time domain signals of a first number of existing audio frames in a case that an audio packet loss occurs;

carrying out acoustic feature extraction on the time domain signals of the first number of the existing audio frames to generate acoustic features of the existing audio frames;

carrying out, by a one-dimensional convolutional network, semantic feature extraction on the time domain signals of the first number of the existing audio frames, and carrying out, by a GRU network, semantic understanding based on the extracted semantic features to generate semantic expressions at a phoneme level corresponding to the existing audio frames;

predicting a time domain signal of a current audio frame according to the acoustic features and the semantic expressions, for audio packet loss compensation; and

performing the audio packet loss compensation using the predicted time domain signal.

17 . The electronic device of claim 16 , wherein the one or more processors are configured to execute operations of carrying out the acoustic feature extraction on the time domain signals of the first number of the existing audio frames to generate the acoustic features of the existing audio frames by:

carrying out, by the one-dimensional convolutional network with multiple branches, acoustic feature extraction on the time domain signals of the first number of the existing audio frames to generate multiple low-dimensional acoustic features; and

carrying out aggregation processing on the multiple low-dimensional acoustic features to generate the acoustic features of the existing audio frames.

18 . The electronic device of claim 17 , wherein the one or more processors are configured to execute operations of carrying out aggregation processing on the multiple low-dimensional acoustic features to generate acoustic features of the existing audio frames by:

aggregating, by an attention module, the multiple low-dimensional acoustic features to generate the acoustic features of the existing audio frames.