Deep prediction refinement

1. A method, comprising:

obtaining a first motion-compensated region for a block of a picture and obtaining a second region for the block;

providing as input to a main branch of a neural network the first motion-compensated region and the second region;

providing as input to a skip connection of the neural network a tensor that concatenates the first motion-compensated region and the second region;

obtaining from the skip connection a combination of the tensor with an output of a layer of the main branch of the neural network, wherein obtaining the combination includes one of: concatenating the tensor with the output of the layer of the main branch or performing a convolution on the tensor and adding the convoluted tensor to the output of the layer of the main branch;

performing a convolution on the combination obtained from the skip connection;

obtaining a prediction for the block based on an output of the convolution performed on the combination obtained from the skip connection; and

encoding the block based on the prediction.

2. The method of claim 1 , wherein the prediction is output by the Neural Network.

3. The method of claim 1 , wherein obtaining a prediction for the block comprises:

obtaining a map of prediction refinement parameters for the block using the neural network, wherein the neural network uses the first motion-compensated region and the second region; and

obtaining the prediction for the block based at least on the first motion-compensated region and the map of prediction refinement parameters.

4. The method of claim 3 , wherein the neural network comprises one branch per prediction parameter of the map of prediction refinement parameters, and wherein the branch comprises at least one convolutional layer.

5. The method of claim 4 , wherein a different block size is used for each branch of the neural network.

6. The method of claim 1 , wherein the second region comprises a second motion-compensated region for the block.

7. An apparatus, comprising one or more processors, wherein the one or more processors are configured to:

obtain a first motion-compensated region for a block of a picture and obtain a second region for the block;

provide as input to a main branch of a neural network the first motion-compensated region and the second region;

provide as input to a skip connection of the neural network a tensor that concatenates the first motion-compensated region and the second region;

obtain from the skip connection a combination of the tensor with an output of a layer of the main branch of the neural network, wherein obtaining the combination includes one of: concatenating the tensor with the output of the layer of the main branch or performing a convolution on the tensor and adding the convoluted tensor to the output of the layer of the main branch;

perform a convolution on the combination obtained from the skip connection;

obtain a prediction for the block based on an output of the convolution performed on the combination obtained from the skip connection; and

encode the block based on the prediction.

8. The apparatus of claim 7 , wherein the neural network comprises a set of convolutional layers, and wherein a number of convolutions for each layer is a multiple of a power of 2.

9. A method, comprising:

obtaining a first motion-compensated region for a block of a picture and obtaining a second region for the block;

providing as input to a main branch of a neural network the first motion-compensated region and the second region;

providing as input to a skip connection of the neural network a tensor that concatenates the first motion-compensated region and the second region;

obtaining from the skip connection a combination of the tensor with an output of a layer of the main branch of the neural network, wherein obtaining the combination includes one of: concatenating the tensor with the output of the layer of the main branch or performing a convolution on the tensor and adding the convoluted tensor to the output of the layer of the main branch;

performing a convolution on the combination obtained from the skip connection;

obtaining a prediction for the block based on an output of the convolution performed on the combination obtained from the skip connection; and

decoding the block based on the prediction.

10. The method of claim 9 , wherein additional data is provided to the neural network, wherein the additional data comprises at least one of the following:

an information representative of a filter used for motion compensation;

an information representative of a quantization parameter used for encoding the block; or

an information representative of at least one motion field determined for the block.

11. The method of claim 9 , wherein the first motion-compensated region and the second region are respectively enlarged according to at least a size of a receptive field of the neural network before being used by the neural network.

12. An apparatus, comprising one or more processors, wherein the one or more processors are configured to:

obtain a first motion-compensated region for a block of a picture and obtain a second region for the block;

provide as input to a main branch of a neural network the first motion-compensated region and the second region;

provide as input to a skip connection of the neural network a tensor that concatenates the first motion-compensated region and the second region;

obtain from the skip connection a combination of the tensor with an output of a layer of the main branch of the neural network, wherein obtaining the combination includes one of: concatenating the tensor with the output of the layer of the main branch or performing a convolution on the tensor and adding the convoluted tensor to the output of the layer of the main branch;

perform a convolution on the combination obtained from the skip connection;

obtain a prediction for the block based on an output of the convolution performed on the combination obtained from the skip connection; and

decode the block based on the prediction.

13. The apparatus of claim 12 , wherein in obtaining the combination, the layer of the main branch is a last convolutional layer of the neural network.

14. The apparatus of claim 12 , wherein the layer of the main branch has a number N−2 of convolutions where N is a multiple of a power of 2.

15. The apparatus of claim 12 , wherein the output of the layer of the main branch is split in a first part having a size corresponding to a size of an input of the skip connection and a second part, the first part being added to the input of the skip connection, concatenated with the second part and provided to a next layer of the neural network.

16. The apparatus of claim 12 , further comprising at least one of: an antenna configured to receive a signal, the signal including data representative of the picture; a band limiter configured to limit the received signal to a band of frequencies that includes the data representative of the picture; or a display configured to display the picture.

17. The apparatus of claim 16 , further comprising a television, a cell phone, a tablet, or a set top box.

18. The apparatus of claim 12 , wherein additional data is provided to the neural network, and wherein the additional data comprises at least one of the following:

an information representative of a filter used for motion compensation;

an information representative of a quantization parameter used for encoding the block; or

an information representative of at least one motion field determined for the block.

19. A computer readable storage medium having stored thereon instructions for causing one or more processors to perform a method comprising:

obtaining a first motion-compensated region for a block of a picture and obtaining a second region for the block;

providing as input to a main branch of a neural network the first motion-compensated region and the second region;

providing as input to a skip connection of the neural network a tensor that concatenates the first motion-compensated region and the second region;

obtaining from the skip connection a combination of the tensor with an output of a layer of the main branch of the neural network, wherein obtaining the combination includes one of: concatenating the tensor with the output of the layer of the main branch or performing a convolution on the tensor and adding the convoluted tensor to the output of the layer of the main branch;

performing a convolution on the combination obtained from the skip connection;

obtaining a prediction for the block based on an output of the convolution performed on the combination obtained from the skip connection; and

encoding the block based on the prediction.

20. A computer readable storage medium having stored thereon instructions for causing one or more processors to perform a method comprising:

obtaining a first motion-compensated region for a block of a picture and obtaining a second region for the block;

providing as input to a main branch of a neural network the first motion-compensated region and the second region;

providing as input to a skip connection of the neural network a tensor that concatenates the first motion-compensated region and the second region;

obtaining from the skip connection a combination of the tensor with an output of a layer of the main branch of the neural network, wherein obtaining the combination includes one of: concatenating the tensor with the output of the layer of the main branch or performing a convolution on the tensor and adding the convoluted tensor to the output of the layer of the main branch;

performing a convolution on the combination obtained from the skip connection;

obtaining a prediction for the block based on an output of the convolution performed on the combination obtained from the skip connection; and

decoding the block based on the prediction.