Deep learning based image partitioning for video compression
A block of video data is split using one or more of several possible partition operations by using the partitioning choices obtained through use of a deep learning-based image partitioning. In at least one embodiment, the block is split in one or more splitting operations using a convolutional neural network. In another embodiment, inputs to the convolutional neural network come from pixels along the block's causal borders. In another embodiment, boundary information, such as the location of partitions in spatially neighboring blocks, is used by the convolutional neural network. Methods, apparatus, and signal embodiments are provided for encoding.
1. A method for video processing, comprising:
determining, using a convolutional neural network, a vector of split possibilities based on a block of image data and a plurality of pixels adjacent to the block, wherein the plurality of pixels comprises a causal pixel, wherein the causal pixel is adjacent to the block of image data, wherein the causal pixel is a pixel of a causal border of the block of image data, wherein the causal border comprises a row of pixels on top of the block of image data and a column of pixels to the left of the block of image data, wherein the convolutional neural network comprises a convolutional layer and a fully connected layer, wherein an output of the convolutional layer comprises a vector associated with a level of splitting, wherein the output of the convolutional layer is concatenated with quantization information associated with the block of image data, wherein the output of the convolution layer, concatenated with the quantization information, is an input of the fully connected layer, and wherein an output of the fully connected layer, based on a dimension reduction, comprises the vector of split possibilities;
partitioning the block of image data into a plurality of smaller blocks based on the vector of split possibilities; and
predicting the plurality of smaller blocks.
2. A non-transitory computer readable medium containing program instructions that, when executed by a processor, cause the processor to perform a method comprising:
determining, using a convolutional neural network, a vector of split possibilities based on a block of image data and a plurality of pixels adjacent to the block, wherein the plurality of pixels comprises a causal pixel, wherein the causal pixel is adjacent to the block of image data, wherein the causal pixel is a pixel of a causal border of the block of image data, wherein the causal border comprises a row of pixels on top of the block of image data and a column of pixels to the left of the block of image data, wherein the convolutional neural network comprises a convolutional layer and a fully connected layer, wherein an output of the convolutional layer comprises a vector associated with a level of splitting, wherein the output of the convolutional layer is concatenated with quantization information associated with the block of image data, wherein the output of the convolution layer, concatenated with the quantization information, is an input of the fully connected layer, and wherein an output of the fully connected layer, based on a dimension reduction, comprises the vector of split possibilities;
partitioning the block of image data into a plurality of smaller blocks based on the vector of split possibilities; and
predicting the plurality of smaller blocks.
3. A non-transitory computer program product comprising program instructions which, when executed by a computer, cause the computer to decode or encode data generated according to the method of claim 1 .
4. An apparatus for video processing, comprising:
a memory, and
at least one processor, configured to:
determine, using a convolutional neural network, a vector of split possibilities based on a block of image data and a plurality of pixels adjacent to the block, wherein the plurality of pixels comprises a causal pixel, wherein the causal pixel is adjacent to the block of image data, wherein the causal pixel is a pixel of a causal border of the block of image data, wherein the causal border comprises a row of pixels on top of the block of image data and a column of pixels to the left of the block of image data, wherein the convolutional neural network comprises a convolutional layer and a fully connected layer, wherein an output of the convolutional layer comprises a vector associated with a level of splitting, wherein the output of the convolutional layer is concatenated with quantization information associated with the block of image data, wherein the output of the convolutional layer, concatenated with the quantization information, is an input of the fully connected layer, and wherein an output of the fully connected layer, based on a dimension reduction, comprises the vector of split possibilities;
partition the block of image data into a plurality of smaller blocks based on the vector of split possibilities; and
predict the plurality of smaller blocks.
5. A non-transitory computer program product comprising program instructions which, when executed by a processor, cause the processor to decode or encode data generated by the apparatus of claim 4 .
6. The method of claim 1 , wherein the quantization information comprises a quantization parameter of the block of image data.
7. The apparatus of claim 4 , wherein the quantization information comprises a quantization parameter of the block of image data.
8. The method of claim 1 , wherein the causal border of the block of image data is associated with a reconstructed image.
9. The method of claim 1 , wherein information relating to a spatial neighboring block of the block of image data is concatenated to an input and further input to said convolutional neural network.
10. The apparatus of claim 4 , wherein the block of image data is associated with an image, and the causal border of the block of image data is associated with the image.
11. The apparatus of claim 4 , wherein the causal border of the block of image data is associated with a reconstructed image.
12. The apparatus of claim 4 , wherein information relating to a spatial neighboring block of the block of image data is concatenated to an input and further input to the convolutional neural network.
13. The apparatus of claim 4 , wherein the determination of the vector of split possibilities using the convolutional neural network is based on location information of a partition boundary associated with a neighboring block, and at least one vector representing the location information of the partition boundary associated with the neighboring block is reduced in dimension for use in the convolutional neural network, and wherein the neighboring block is located above or to the left of the block of image data.
14. The apparatus of claim 13 , wherein the location information of the partition boundary associated with the neighboring block is contained in a vector.
15. The apparatus of claim 4 , wherein the at least one processor is further configured to:
generate a residual based on the predicted plurality of smaller blocks; and
include the residual in a bitstream.
16. The apparatus of claim 4 , wherein the vector of split possibilities is a single-component vector.
17. The apparatus of claim 4 , wherein the block of image data has a size of 64×64 pixels.
18. The method of claim 1 , wherein the vector of split possibilities is a single-component vector.
19. The method of claim 1 , wherein the block of image data has a size of 64×64 pixels.