Neural network-based intra prediction for video encoding or decoding
A video coding system is provided that performs intra prediction in a mode using a neural network for block of only a set of specific block sizes. The signaling of this mode is designed to be efficient in terms of rate-distortion under this constraint. Different transformations of the context of a block and the neural network prediction of this block are introduced in order to use one single neural network for predicting blocks of several sizes, as well as the corresponding signaling. The neural network-based prediction mode considers both luminance blocks and chrominance blocks. The video coding system comprises encoder and decoder apparatuses, encoding, decoding and signal generation methods and a signal carrying information corresponding to the described coding mode.
1. A video encoding method comprising:
performing intra prediction for at least one block in a picture or video using a neural network based intra prediction by feeding a block context into a neural network selected based on a size of the at least one block, the block context comprising pixels of blocks located at a top side, at a left side, at a diagonal top left side, at a diagonal top right side and at a diagonal bottom left side of the at least one block, wherein the size of the block context is based on the size of the at least one block, and wherein the block context comprises n l columns and n a rows, wherein n l and n a are selected as:
n
a
=
α
H
,
n
l
=
β
W
,
α
∈
〚
1
4
,
1
2
,
3
4
,
1
,
2
〛
,
β
∈
〚
1
4
,
1
2
,
3
4
,
1
,
2
〛
where H is a height of the block and W is a width of the block;
generating signaling information representative that the intra prediction mode is a neural network based intra prediction; and
encoding at least information representative of the at least one block and the neural network-based intra prediction mode.
2. The method of claim 1 , wherein the signaling information is encoded in a bitstream and comprises a flag indicating that a neural network-based intra prediction mode is selected for the at least one block, the flag being based on a set of flags representing a plurality of intra prediction modes arranged in a binary tree for being encoded in a bitstream and wherein the flag indicating that neural network-based intra prediction mode is selected is located at a first level of the tree and encoded with a single bit.
3. A video decoding method comprising:
obtaining, for at least one block in a picture or video, at least information representative that an intra prediction is a neural network-based prediction and a block context, the block context comprising pixels of blocks located at a top side, at a left side, at a diagonal top left side, at a diagonal top right side and at a diagonal bottom left side of the at least one block, wherein the size of the block context is based on the size of the at least one block, and wherein the block context comprises n l columns and n a rows, wherein n l and n a are selected as:
n
a
=
α
H
,
n
l
=
β
W
,
α
∈
〚
1
4
,
1
2
,
3
4
,
1
,
2
〛
,
β
∈
〚
1
4
,
1
2
,
3
4
,
1
,
2
〛
where H is a height of the block and W is a width of the block; and
performing intra prediction for the at least one block in a picture or video by feeding the block context into a neural network based on the size of the at least one block.
4. The method of claim 3 , wherein the neural network-based intra prediction is performed based on a position of the at least one block.
5. The method of claim 3 , wherein the block context is down-sampled prior to performing the intra prediction and the at least one predicted block resulting from the neural network based intra prediction is interpolated after the intra prediction.
6. The method of claim 3 , wherein the block context is transposed prior to performing the intra prediction and the at least one predicted block resulting from the neural network based intra prediction is transposed back after the intra prediction.
7. The method of claim 3 , wherein the block context is down-sampled and transposed prior to performing the intra prediction and the at least one predicted block resulting from the neural network based intra prediction is transposed back and interpolated after the intra prediction.
8. The method of claim 3 , wherein the neural network-based intra prediction is done in both luminance and chrominance of the at least one block.
9. An apparatus, comprising an encoder for encoding a current block in a picture or video wherein the encoder is configured to:
perform intra prediction for at least one block in a picture or video using a neural network based intra prediction by feeding a block context into a neural network selected based on a size of the at least one block, the block context comprising pixels of blocks located at a top side, at a left side, at a diagonal top left side, at a diagonal top right side and at a diagonal bottom left side of the at least one block, wherein the size of the block context is based on the size of the at least one block, and wherein the block context comprises n l columns and n a rows, wherein n l and n a are selected as:
n
a
=
α
H
,
n
l
=
β
W
,
α
∈
〚
1
4
,
1
2
,
3
4
,
1
,
2
〛
,
β
∈
〚
1
4
,
1
2
,
3
4
,
1
,
2
〛
where H is a height of the block and W is a width of the block;
generate signaling information representative that the intra prediction mode is a neural network based intra prediction; and
encode at least information representative of the at least one block and the neural network-based intra prediction mode.
10. An apparatus, comprising a decoder for decoding picture data for a current block in a picture or video wherein the decoder is configured to:
obtain, for at least one block in a picture or video, at least information representative that an intra prediction is a neural network-based prediction and a block context, the block context comprising pixels of blocks located at a top side, at a left side, at a diagonal top left side, at a diagonal top right side and at a diagonal bottom left side of the at least one block, wherein the size of the block context is based on the size of the at least one block, and wherein the block context comprises n l columns and n a rows, wherein n l and n a are selected as:
n
a
=
α
H
,
n
l
=
β
W
,
α
∈
〚
1
4
,
1
2
,
3
4
,
1
,
2
〛
,
β
∈
〚
1
4
,
1
2
,
3
4
,
1
,
2
〛
where H is a height of the block and W is a width of the block; and
perform intra prediction for the at least one block in a picture or video by feeding the block context into a neural network selected based on the size of the at least one block.
11. The apparatus of claim 10 , wherein the neural network-based intra prediction is performed based on a position of the at least one block.
12. The apparatus of claim 10 , wherein the block context is down-sampled prior to performing the intra prediction and the at least one predicted block resulting from the neural network-based intra prediction is interpolated after the intra prediction.
13. The apparatus of claim 10 , wherein the block context is transposed prior to performing the intra prediction and the at least one predicted block resulting from the neural network-based intra prediction is transposed back after the intra prediction.
14. The apparatus of claim 10 , wherein the block context is down-sampled and transposed prior to performing the intra prediction and the at least one predicted block resulting from the neural network-based intra prediction is transposed back and interpolated after the intra prediction.
15. A non-transitory computer readable medium comprising program code instructions for implementing the steps of a method according to claim 3 when executed by a processor.
16. The method of claim 1 , wherein the neural network is selected among a set of at least two neural networks, and wherein at least one size of a block is associated to each neural network of the set of at least two neural networks.
17. The method of claim 3 , wherein the neural network is selected among a set of at least two neural networks, and wherein at least one size of a block is associated to each neural network of the set of at least two neural networks.
18. The apparatus of claim 9 , wherein the signaling information is encoded in a bitstream and comprises a flag indicating that a neural network-based intra prediction mode is selected for the at least one block, the flag being based on a set of flags representing a plurality of intra prediction modes arranged in a binary tree for being encoded in a bitstream and wherein the flag indicating that neural network-based intra prediction mode is selected is located at a first level of the tree and encoded with a single bit.
19. The apparatus of claim 9 , wherein the neural network is selected among a set of at least two neural networks, and wherein at least one size of a block is associated to each neural network of the set of at least two neural networks.
20. The apparatus of claim 10 , wherein the neural network is selected among a set of at least two neural networks, and wherein at least one size of a block is associated to each neural network of the set of at least two neural networks.