Method and apparatus for video encoding and decoding based on neural network implementation of CABAC
Methods and apparatuses for video coding and decoding are provided. The method of video encoding includes accessing a bin of a syntax element associated with a block in a picture of a video, determining a context for the bin of the syntax element associated with the syntax element and entropy encoding the bin of the syntax element based on the determined context wherein either the bin of the syntax element is based on the relevance of a prediction by a neural network of the syntax element or the probability associated to the context is determined by a neural network. A bitstream formatted to include encoded data, a computer-readable storage medium and a computer-readable program product are also described.
1. A method of video encoding comprising:
accessing a bin of a syntax element associated with a current block in a picture of a video;
determining a context for the bin of the syntax element, said context being associated with said syntax element; and
entropy encoding the bin of the syntax element based on the determined context;
wherein said accessing a bin of a syntax element comprises:
determining, for the syntax element, a prediction of the syntax element by applying a neural network on data encoded prior to said syntax element, said data encoded prior to said syntax element comprising spatial and temporal information relative to a block previously encoded or to the current block, wherein the prediction of the syntax element output from the neural network is a vector giving the probabilities for each of the possible value of syntax element; and
determining a bin of the syntax element based on a value representative of the difference between the prediction and the syntax element, wherein the value representative of the difference between the prediction and the syntax element is a codeword for each of the syntax element values based on the associated probability using a Huffman coding.
2. The method according to claim 1 , wherein said data encoded prior to said syntax element comprises at least one among:
the values of a previously encoded syntax elements of the same type in a neighborhood of the current block;
the reconstructed samples in the L-shape of the current block;
the prediction samples of the current block;
the reconstructed residuals of the current block;
the samples in the L-shape of a reference block for the current block when the current block is coded in inter;
the motion field for the current block when the current block is coded in inter;
the prediction samples of the current block generated with motion compensation using said motion field;
the quantization parameter; or
the current block size.
3. A method of video decoding comprising:
accessing an encoded bin of a syntax element associated with a current block in a picture of an encoded video;
determining a context for the bin of the syntax element associated with said syntax element; and
entropy decoding the encoded bin of the syntax element based on the determined context;
wherein said entropy decoding comprises:
determining, for the syntax element, a prediction of the syntax element by applying a neural network on data decoded prior to said syntax element, said data decoded prior to said syntax element comprising spatial and temporal information relative to a block previously decoded or to the current block, wherein the prediction of the syntax element output from the neural network is a vector giving the probabilities for each of the possible value of syntax element; and
determining the syntax element based on the determined prediction of the syntax element and the decoded bin of the syntax element, the decoded bin of the syntax element being a value representative of the difference between the prediction and the syntax element, wherein the value representative of the difference between the prediction and the syntax element is a codeword for each of the syntax element values based on the associated probability using a Huffman coding.
4. The method according to claim 3 , wherein said data decoded prior to said syntax element comprises at least one among:
the values of a previously decoded syntax elements of the same type in a neighborhood of the current block;
the reconstructed samples in the L-shape of the current block;
the prediction samples of the current block;
the reconstructed residuals of the current block;
the samples in the L-shape of a reference block for the current block when the current block is coded in inter;
the motion field for the current block when the current block is coded in inter;
the prediction samples of the current block generated with motion compensation using said motion field;
the quantization parameter; or
the current block size.
5. An apparatus for video encoding comprising a memory and one or more processors configured to:
access a bin of a syntax element associated with a current block in a picture of a video;
determine a context for the bin of the syntax element associated with said syntax element; and
entropy encode the bin of the syntax element based on the determined context;
wherein said one or more processors are further configured to:
determine by a neural network, for the syntax element, a prediction of the syntax element responsive to data encoded prior to said syntax element, said data encoded prior to said syntax element comprising spatial and temporal information relative to a block previously encoded or to the current block, wherein the prediction of the syntax element output from the neural network is a vector giving the probabilities for each of the possible value of syntax element; and
determine a bin of the syntax element based on a value representative of the difference between the prediction and the syntax element, wherein the value representative of the difference between the prediction and the syntax element is a codeword for each of the syntax element values based on the associated probability using a Huffman coding.
6. The apparatus according to claim 5 , wherein said data encoded prior to said syntax element comprises at least one among:
the values of a previously encoded syntax elements of the same type in a neighborhood of the current block;
the reconstructed samples in the L-shape of the current block;
the prediction samples of the current block;
the reconstructed residuals of the current block;
the samples in the L-shape of a reference block for the current block when the current block is coded in inter;
the motion field for the current block when the current block is coded in inter;
the prediction samples of the current block generated with motion compensation using said motion field;
the quantization parameter; or
the current block size.
7. An apparatus for video decoding comprising a memory and one or more processors configured to:
access an encoded bin of a syntax element associated with a current block in a picture of an encoded video;
determine a context for the bin of the syntax element associated with said syntax element; and
entropy decode the encoded bin of syntax element based on the determined context;
wherein said one or more processors are further configured to:
determine, by a neural network, for the syntax element, a prediction of the syntax element responsive to data decoded prior to said syntax element, said data decoded prior to said syntax element comprising spatial and temporal information relative to a block previously decoded or to the current block, wherein the prediction of the syntax element output from the neural network is a vector giving the probabilities for each of the possible value of syntax element;
determine the syntax element responsive to the determined prediction of the syntax element and the decoded bin of the syntax element, the decoded bin of the syntax element being a value representative of the difference between the prediction and the syntax element, wherein the value representative of the difference between the prediction and the syntax element is a codeword for each of the syntax element values based on the associated probability using a Huffman coding.
8. The device according to claim 7 , wherein said data decoded prior to said syntax element comprises at least one among:
the values of a previously decoded syntax elements of the same type in a neighborhood of the current block;
the reconstructed samples in the L-shape of the current block;
the prediction samples of the current block;
the reconstructed residuals of the current block;
the samples in the L-shape of a reference block for the current block when the current block is coded in inter;
the motion field for the current block when the current block is coded in inter;
the prediction samples of the current block generated with motion compensation using said motion field;
the quantization parameter; or
the current block size.
9. A non-transitory program storage device, readable by a computer, tangibly embodying a program of instructions executable by the computer for performing the method according to claim 1 .
10. A non-transitory program storage device, readable by a computer, tangibly embodying a program of instructions executable by the computer for performing the method according to claim 3 .
11. A non-transitory program storage device having encoded data representative of a block in a picture of a video, the encoded data being generated according to claim 1 .
12. A method of video decoding comprising:
accessing an encoded bin of a syntax element associated with a current block in a picture of an encoded video;
determining a context for the bin of the syntax element associated with said syntax element; and
entropy decoding the encoded bin of the syntax element based on the determined context;
wherein said entropy decoding comprises:
determining, for the syntax element, a prediction of the syntax element by applying a neural network on data decoded prior to said syntax element, said data decoded prior to said syntax element comprising spatial and temporal information relative to a block previously decoded or to the current block; and
determining the syntax element based on the determined prediction of the syntax element and the decoded bin of the syntax element, the decoded bin of the syntax element being a value representative of the difference between the prediction and the syntax element, wherein the syntax element is a flag, and the value representative of the difference between the prediction and the syntax element is a XOR value between the prediction and the flag.
13. An apparatus for video decoding, comprising:
a memory; and
one or more processors configured to:
access an encoded bin of a syntax element associated with a current block in a picture of an encoded video;
determine a context for the bin of the syntax element associated with said syntax element; and
entropy decode the encoded bin of syntax element based on the determined context;
wherein said one or more processors are further configured to:
determine, by a neural network, for the syntax element, a prediction of the syntax element responsive to data decoded prior to said syntax element, said data decoded prior to said syntax element comprising spatial and temporal information relative to a block previously decoded or to the current block;
determine the syntax element responsive to the determined prediction of the syntax element and the decoded bin of the syntax element, the decoded bin of the syntax element being a value representative of the difference between the prediction and the syntax element, wherein the syntax element is a flag, and the value representative of the difference between the prediction and the syntax element is a XOR value between the prediction and the flag.