Granted Patent B2

US 12,739,382 · App. 18/368,683 · Granted Sep 15, 2026

Inventors: Xiang Ma (Moscow, RU); Jue Mao (Hangzhou, CN); Haitao Yang (Shenzhen, CN)

Assignee: HUAWEI TECHNOLOGIES CO., LTD.

H04N19/13H04N19/132H04N19/172H04N19/91

Abstract

Claims (107)

1 . A video picture decoding method, comprising:

receiving a bitstream of a current picture;

performing a probability estimation on input data by using a neural network obtained through training, to obtain a probability distribution of residual values of a plurality of samples comprised in a residual of the current picture, wherein the input data comprises at least a residual of a reference picture, and wherein the reference picture is a decoded picture obtained before the current picture is decoded;

performing an arithmetic entropy decoding on the bitstream based on the probability distribution of the residual values of the plurality of samples comprised in the residual of the current picture, to obtain first entropy decoding data, wherein the first entropy decoding data represents the residual of the current picture; and

obtaining a reconstructed sample value of the current picture based on the residual of the current picture.

2 . The method according to claim 1 , wherein the neural network is trained via a training process, including:

obtaining the input data for the training, wherein the input data for the training comprises a decoded picture obtained before a sample picture is decoded, and wherein the sample picture is a known decoded picture;

performing, by using the neural network, a probability estimation on the input data for the training, to obtain estimated probability distribution of residual values of a plurality of samples comprised in a residual of the sample picture;

obtaining a loss value based on known probability distribution of the residual values of the plurality of samples comprised in the residual of the sample picture and the estimated probability distribution; and

when the loss value converges to a first threshold, or a current quantity of training times of the neural network is greater than or equal to a second threshold, determining that a current neural network is a corresponding neural network obtained after the training of the neural network is completed.

3 . The method according to claim 2 , wherein the input data for the training comprises:

residual values of a plurality of samples comprised in a residual of a first reference picture;

residual values of a plurality of samples comprised in a residual of the first reference picture and residual values of a plurality of samples comprised in a residual of a second reference picture;

predicted values of a plurality of samples comprised in a prediction of the sample picture, residual values of a plurality of samples comprised in a residual of the first reference picture, and predicted values of a plurality of samples comprised in a prediction of the first reference picture; or

predicted values of a plurality of samples comprised in a prediction of the sample picture, residual values of a plurality of samples comprised in a residual of the first reference picture, predicted values of a plurality of samples comprised in a prediction of the first reference picture, predicted values of a plurality of samples comprised in a prediction of a second reference picture, and residual values of a plurality of samples comprised in a residual of the second reference picture, wherein

the first reference picture and the second reference picture are decoded pictures obtained before the sample picture is decoded.

4 . The method according to claim 1 , wherein

the probability distribution represents probability distribution of a plurality of differences between reconstructed values of the plurality of samples comprised in the residual of the current picture and predicted values of the plurality of samples;

the probability distribution represents probability distribution of residual values of all samples in a plurality of samples of the current picture; or

the probability distribution represents the probability distribution of the residual values of the plurality of samples of the current picture.

5 . The method according to claim 1 , wherein the first entropy decoding data is the residual of the current picture, or a feature map of the residual of the current picture, or a transformed and quantized residual of the current picture.

6 . The method according to claim 5 , wherein the first entropy decoding data is the feature map of the residual of the current picture, the method further comprising:

obtaining the residual of the current picture based on the feature map of the residual of the current picture by using a decoder network.

7 . The method according to claim 5 , wherein the first entropy decoding data is the transformed and quantized residual of the current picture, the method further comprising:

performing inverse transformation and inverse quantization on the transformed and quantized residual of the current picture, to obtain the residual of the current picture.

8 . The method according to claim 6 , wherein the input data further comprises hyper prior information;

wherein the neural network comprises a convolutional network, an average pooling layer, a first convolutional layer, a concatenate concat layer, and a normalized exponential function Softmax layer, and wherein the convolutional network comprises a plurality of convolutional layers and an activation ReLU layer; and

wherein the performing the probability estimation on input data by using a neural network, to obtain probability distribution of a residual of the current picture comprises:

separately extracting, by using the convolutional network, feature information from the residual of the reference picture and the hyper prior information; and performing, by using the concatenate concat layer, the first convolutional layer, the average pooling layer, and the Softmax layer, probability estimation on the feature information extracted by the convolutional network, and outputting the probability distribution of the residual values of the plurality of samples and comprised in the residual of the current picture.

9 . The method according to claim 1 , wherein the input data further comprises a prediction of the reference picture and a prediction of the current picture;

wherein the neural network comprises a convolutional network, an average pooling layer, and a normalized exponential function Softmax layer, and wherein the convolutional network comprises a plurality of convolutional layers and an activation ReLU layer; and

wherein the performing the probability estimation on input data by using a neural network, to obtain probability distribution of residual values of a plurality of samples comprised in a residual of the current picture comprises:

extracting, by using the convolutional network, feature information of the residual of the reference picture, the prediction of the reference picture, and the prediction of the current picture, performing, by using the average pooling layer and the Softmax layer, probability estimation on the feature information extracted by the convolutional network, and outputting the probability distribution of the residual values of the plurality of samples comprised in the residual of the current picture.

10 . The method according to claim 1 , wherein the input data comprises:

residuals of a plurality of reference pictures, wherein the plurality of reference pictures are a plurality of decoded pictures obtained before the current picture is decoded;

wherein the neural network comprises a convolutional network, an average pooling layer, and a normalized exponential function Softmax layer, and wherein the convolutional network comprises a plurality of convolutional layers and an activation ReLU layer; and

wherein the performing the probability estimation on input data by using a neural network, to obtain probability distribution of residual values of a plurality of samples comprised in a residual of the current picture comprises:

extracting, by using the convolutional network, feature information of the residuals of the plurality of reference pictures, performing, by using the average pooling layer and the Softmax layer, probability estimation on the feature information extracted by the convolutional network, and outputting the probability distribution of the residual values of the plurality of samples comprised in the residual of the current picture.

11 . The method according to claim 1 , wherein the input data comprises:

residuals of a plurality of reference pictures, predictions of the plurality of reference pictures, and a prediction of the current picture, wherein the plurality of reference pictures are a plurality of decoded pictures obtained before the current picture is decoded;

wherein the neural network comprises a convolutional network, an average pooling layer, and a normalized exponential function Softmax layer, and wherein the convolutional network comprises a plurality of convolutional layers and an activation ReLU layer; and

wherein the performing the probability estimation on input data by using a neural network, to obtain probability distribution of residual values of a plurality of samples comprised in a residual of the current picture comprises:

extracting, by using the convolutional network, feature information of the residuals of the plurality of reference pictures, the predictions of the plurality of reference pictures, and the prediction of the current picture, performing, by using the average pooling layer and the Softmax layer, probability estimation on the feature information extracted by the convolutional network, and outputting the probability distribution of the residual values of the plurality of samples comprised in the residual of the current picture.

12 . A video picture encoding method, comprising:

obtaining a residual of a current picture having a plurality of samples, wherein the residual is differences between original values of the samples and predicted values of the samples;

performing a probability estimation on input data by using a neural network obtained through training, to obtain probability distribution of residual values of the plurality of samples comprised in the residual of the current picture, wherein the input data comprises at least a residual of a reference picture, and wherein the reference picture is an encoded picture obtained before the current picture is encoded; and

performing an arithmetic entropy encoding on first entropy encoding data based on the probability distribution of the residual values of the plurality of samples comprised in the residual of the current picture, to obtain a bitstream of the current picture, wherein the first entropy encoding data represents the residual of the current picture.

13 . The method according to claim 12 , wherein the neural network is trained via a training process, including:

obtaining the input data for the training, wherein the input data for the training comprises an encoded picture obtained before a sample picture is encoded, and the sample picture is a known encoded picture;

performing, by using the neural network, probability estimation on the input data for the training, to obtain estimated probability distribution of residual values of a plurality of samples comprised in a residual of the sample picture;

obtaining a loss value based on known probability distribution of the residual values of the plurality of samples comprised in the residual of the sample picture and the estimated probability distribution; and

when the loss value converges to a first threshold, or a current quantity of training times of the neural network is greater than or equal to a second threshold, determining that a current neural network is a corresponding neural network obtained after the training of the neural network is completed.

14 . The method according to claim 13 , wherein the input data for the training comprises:

residual values of a plurality of samples comprised in a residual of a first reference picture;

residual values of a plurality of samples comprised in a residual of the first reference picture and residual values of a plurality of samples comprised in a residual of a second reference picture;

predicted values of a plurality of samples comprised in a prediction of the sample picture, residual values of a plurality of samples comprised in a residual of the first reference picture, and predicted values of a plurality of samples comprised in a prediction of the first reference picture; or

predicted values of a plurality of samples comprised in a prediction of the sample picture, residual values of a plurality of samples comprised in a residual of the first reference picture, predicted values of a plurality of samples comprised in a prediction of the first reference picture, predicted values of a plurality of samples comprised in a prediction of a second reference picture, and residual values of a plurality of samples comprised in a residual of the second reference picture, wherein

the first reference picture and the second reference picture are encoded pictures obtained before the sample picture is encoded.

15 . The method according to claim 12 , wherein

the probability distribution represents probability distribution of a plurality of differences between original values of the plurality of samples comprised in the residual of the current picture and predicted values of the plurality of samples;

the probability distribution represents probability distribution of residual values of all sample points in a plurality of samples of the current picture; or

the probability distribution represents the probability distribution corresponding to the residual values of the plurality of samples of the current picture.

16 . The method according to claim 12 , wherein the first entropy encoding data comprises the residual of the current picture, or a feature map of the residual of the current picture, or a transformed and quantized residual of the current picture.

17 . The method according to claim 16 , wherein the first entropy encoding data is the feature map of the residual of the current picture, the method further comprising:

obtaining the feature map of the residual of the current picture from the residual of the current picture by using an encoder network.

18 . The method according to claim 16 , wherein the first entropy encoding data is the transformed and quantized residual of the current picture, the method further comprising:

transforming and quantizing the residual of the current picture to obtain a transformed and quantized residual of the current picture.

19 . The method according to claim 17 , wherein

the input data further comprises hyper prior information;

wherein the neural network comprises a convolutional network, an average pooling layer, a first convolutional layer, a concatenate concat layer, and a normalized exponential function Softmax layer, and wherein the convolutional network comprises a plurality of convolutional layers and an activation ReLU layer; and

wherein the performing the probability estimation on input data by using a neural network, to obtain probability distribution of residual values of a plurality of samples comprised in a residual of the current picture comprises:

extracting, by using the convolutional network, feature information from the residual of the reference picture and the hyper prior information; and performing, by using the average pooling layer, the first convolutional layer, the concat layer, and the Softmax layer, probability estimation on the feature information extracted by the convolutional network, and outputting the probability distribution of the residual values of the plurality of samples and comprised in the residual of the current picture.

20 . The method according to claim 19 , further comprising:

inputting the feature map of the residual of the current picture into a hyper prior encoder network, to obtain a feature vector of the hyper prior information;

rounding the feature vector of the hyper prior information to obtain a rounded feature vector of the hyper prior information; and

inputting the rounded feature vector of the hyper prior information into a hyper prior decoder network, to obtain the hyper prior information.

21 . The method according to claim 12 , wherein

the input data further comprises a prediction of the reference picture and a prediction of the current picture;

wherein the neural network comprises a convolutional network, an average pooling layer, and a normalized exponential function Softmax layer, and wherein the convolutional network comprises a plurality of convolutional layers and an activation ReLU layer; and

wherein the performing the probability estimation on input data by using a neural network, to obtain probability distribution of residual values of a plurality of samples comprised in a residual of the current picture comprises:

extracting, by using the convolutional network, feature information of the residual of the reference picture, the prediction of the reference picture, and the prediction of the current picture, performing, by using the average pooling layer and the Softmax layer, probability estimation on the feature information extracted by the convolutional network, and outputting the probability distribution of the residual values of the plurality of samples of the current picture.

22 . The method according to claim 12 , wherein

the input data comprises residuals of a plurality of reference pictures, prediction signals of the plurality of reference pictures, and a prediction signal of the current picture, wherein the plurality of reference pictures are a plurality of encoded pictures obtained before the current picture is encoded;

wherein the neural network comprises a convolutional network, an average pooling layer, and a normalized exponential function Softmax layer, and wherein the convolutional network comprises a plurality of convolutional layers and an activation ReLU layer; and

wherein the performing the probability estimation on input data by using a neural network, to obtain probability distribution of residual values of a plurality of samples and comprised in a residual of the current picture comprises:

extracting, by using the convolutional network, feature information of the residuals of the plurality of reference pictures, the predictions of the plurality of reference pictures, and the prediction of the current picture, performing, by using the average pooling layer and the Softmax layer, probability estimation on the feature information extracted by the convolutional network, and outputting the probability distribution of the residual values of the plurality of samples comprised in the residual of the current picture.

23 . The method according to claim 12 , wherein

the input data comprises residuals of a plurality of reference pictures, wherein the plurality of reference pictures are a plurality of encoded pictures obtained before the current picture is encoded;

wherein the neural network comprises a convolutional network, an average pooling layer, and a normalized exponential function Softmax layer, and wherein the convolutional network comprises a plurality of convolutional layers and an activation ReLU layer; and

wherein the performing the probability estimation on input data by using a neural network, to obtain probability distribution of residual values of a plurality of samples comprised in a residual of the current picture comprises:

extracting, by using the convolutional network, feature information of the residuals of the plurality of reference pictures, performing, by using the average pooling layer and the Softmax layer, probability estimation on the feature information extracted by the convolutional network, and outputting the probability distribution of the residual values of the plurality of samples comprised in the residual of the current picture.

24 . A decoder, comprising:

a processing circuit; and

a memory coupled to the processing circuit to store instructions, which when executed by the processing circuit, cause the processing circuit to

receive a bitstream of a current picture;

perform a probability estimation on input data by using a neural network obtained through training, to obtain a probability distribution of residual values of a plurality of samples comprised in a residual of the current picture, wherein the input data comprises at least a residual of a reference picture, and wherein the reference picture is a decoded picture obtained before the current picture is decoded;

perform an arithmetic entropy decoding on the bitstream based on the probability distribution of the residual values of the plurality of samples comprised in the residual of the current picture, to obtain first entropy decoding data, wherein the first entropy decoding data represents the residual of the current picture; and

obtain a reconstructed sample value of the current picture based on the residual of the current picture.

25 . An encoder, comprising:

a processing circuit; and

a memory coupled to the processing circuit to store instructions, which when executed by the processing circuit, cause the processing circuit to perform the method according to claim 12 .

26 . A decoder, comprising:

one or more processors; and

a non-transitory computer-readable storage medium, coupled to the processor and storing a program for execution by the processor, wherein when executed by the processor, the program enables the decoder to perform the method according to claim 1 .

27 . An encoder, comprising:

one or more processors; and

a non-transitory computer-readable storage medium, coupled to the processor and storing a program for execution by the processor, wherein when executed by the processor, the program enables the decoder to perform the method according to claim 12 .

Assignments (1)

Priority Claims (1)

Continuity (2)

References Cited (10)