Multiple neural network models for filtering during video coding
An example device for filtering decoded video data includes one or more processors configured to execute a neural network filtering unit to: receive data from one or more other units of the device, the data from the one or more other units of the device being different than data for a decoded picture of video data, and wherein to receive the data from the one or more other units of the device, the one or more processors are configured to execute the neural network filtering unit to receive boundary strength data from a deblocking unit of the device; determine one or more neural network models to be used to filter a portion of the decoded picture; and filter the portion of the decoded picture using the one or more neural network models and the data from the one or more other units of the device, including the boundary strength data.
1. A method of filtering decoded video data, the method comprising:
receiving, by a neural network filtering unit of a video decoding device, data for a decoded picture of video data;
receiving, by the neural network filtering unit, data from one or more other units of the video decoding device, the data from the one or more other units being different than the data for the decoded picture, and wherein receiving the data from the one or more other units of the video decoding device comprises receiving boundary strength data from a deblocking unit of the video decoding device;
determining, by the neural network filtering unit, one or more neural network models to be used to filter a portion of the decoded picture; and
filtering, by the neural network filtering unit, the portion of the decoded picture using the one or more neural network models and the data from the one or more other units of the video decoding device, including the boundary strength data.
2. The method of claim 1 , wherein receiving the data from the one or more other units of the video decoding device further comprises receiving the data from one or more of:
an intra-prediction unit of the video decoding device;
an inter-prediction unit of the video decoding device;
a transform processing unit of the video decoding device;
a quantization unit of the video decoding device;
a loop filter unit of the video decoding device;
a pre-processing unit of the video decoding device; or
a second neural network filtering unit of the video decoding device.
3. The method of claim 2 , wherein the loop filter unit comprises at least one of a sample adaptive offset (SAO) filtering unit or an adaptive loop filtering (ALF) unit.
4. The method of claim 1 , wherein receiving the data further comprises receiving one or more of coding unit (CU) partitioning data, prediction unit (PU) partitioning data, transform unit (TU) partitioning data, deblocking filtering data, quantization parameter (QP) data, intra-prediction data, inter-prediction data, data representing distance between the decoded picture and one or more reference pictures, or motion information for one or more decoded blocks of the decoded picture.
5. The method of claim 4 , wherein the deblocking filtering data includes one or more of whether long or short filters were used for deblocking or whether strong or weak filters were used for deblocking.
6. The method of claim 4 , wherein the intra-prediction data includes an intra-prediction mode.
7. The method of claim 4 , wherein the data representing the distance comprises data representing a difference between a picture order count (POC) value for the decoded picture and a POC value for a reference picture used to predict a block of the decoded picture.
8. The method of claim 1 , wherein filtering the portion of the decoded picture using the one or more neural network models and the data from the one or more other units of the video decoding device comprises providing the data from the one or more other units of the video decoding device as one or more additional input planes to a convolutional neural network (CNN).
9. The method of claim 1 , wherein filtering the portion of the decoded picture using the one or more neural network models and the data from the one or more other units of the video decoding device comprises:
combining a plurality of input planes into a combined input plane, including, for each position (i, j) of the plurality of input planes, setting a value for position (i, j) of the combined input plane equal to a maximum of the values at position (i, j) of the plurality of input planes; and
providing the combined input plane to a convolutional neural network (CNN).
10. The method of claim 1 , wherein filtering the portion of the decoded picture using the one or more neural network models and the data from the one or more other units of the video decoding device comprises adjusting output of the one or more neural network models using the data from the one or more other units of the video decoding device.
11. The method of claim 1 , further comprising adjusting the data from the one or more other units of the video decoding device prior to filtering the portion of the decoded picture.
12. The method of claim 11 , wherein adjusting the data comprises converting values of the data between integer representation and floating point representation.
13. The method of claim 11 , wherein adjusting the data comprises scaling values of the data to be within a range of values suitable for the one or more neural network models.
14. The method of claim 1 , wherein receiving the data comprises receiving partition data for the decoded picture, and wherein filtering the portion of the decoded picture using the one or more neural network models and the data from the one or more other units of the video decoding device comprises:
setting values at positions in an input plane collocated with positions of boundary samples defining partition boundaries in the decoded picture, as indicated by the partition data, to a first value;
setting values at positions in the input plane collocated with positions of internal samples that are non-boundary samples to a second value; and
filtering the portion of the decoded picture using the input plane as an input to at least one of the one or more neural network models.
15. The method of claim 14 , wherein the first value comprises 1 and the second value comprises 0.
16. The method of claim 14 , wherein the partition data comprises coding unit (CU) partition data and the input plane comprises a first partition plane, the method further comprising:
receiving prediction unit (PU) partition data;
forming a second input plane using the PU partition data;
receiving transform unit (TU) partition data; and
forming a third input plane using the TU partition data,
wherein filtering the portion of the decoded picture using the one or more neural network models and the data from the one or more other units of the video decoding device comprises filtering the portion of the decoded picture using the first partition plane, the second input plane, and the third input plane as inputs to at least one of the one or more neural network models.
17. The method of claim 1 , wherein filtering the portion of the decoded picture using the one or more neural network models and the data from the one or more other units of the video decoding device comprises:
converting deblocking filter data for the decoded picture from the deblocking unit to one or more input planes for at least one of the one or more neural network models; and
filtering the portion of the decoded picture using the one or more input planes as inputs to the at least one of the one or more neural network models.
18. The method of claim 1 , further comprising:
encoding a current picture; and
decoding the current picture to form the decoded picture.
19. The method of claim 18 , wherein determining the one or more neural network models comprises determining the one or more neural network models according to a rate-distortion computation.
20. The method of claim 1 , wherein the boundary strength data indicates that a boundary strength value is zero.
21. The method of claim 1 , wherein the boundary strength data indicates that a boundary strength value is one or two.
22. A device for filtering decoded video data, the device comprising:
a memory configured to store a decoded picture of video data; and
one or more processors implemented in circuitry and configured to execute a neural network filtering unit to:
receive data from one or more other units of the device, the data from the one or more other units of the device being different than data for the decoded picture, and wherein to receive the data from the one or more other units of the device, the one or more processors are configured to execute the neural network filtering unit to receive boundary strength data from a deblocking unit of the device;
determine one or more neural network models to be used to filter a portion of the decoded picture; and
filter the portion of the decoded picture using the one or more neural network models and the data from the one or more other units of the device, including the boundary strength data.
23. The device of claim 22 , wherein to receive the data from the one or more other units of the device, the one or more processors are further configured to execute the neural network filtering unit to receive data from one or more of:
an intra-prediction unit of the device;
an inter-prediction unit of the device;
a transform processing unit of the device;
a quantization unit of the device;
a loop filter unit of the device;
a pre-processing unit of the device; or
a second neural network filtering unit of the device.
24. The device of claim 22 , wherein to receive the data from the one or more other units of the device, the one or more processors are further configured to execute the neural network filtering unit to receive one or more of coding unit (CU) partitioning data, prediction unit (PU) partitioning data, transform unit (TU) partitioning data, deblocking filtering data, quantization parameter (QP) data, intra-prediction data, inter-prediction data, data representing distance between the decoded picture and one or more reference pictures, or motion information for one or more decoded blocks of the decoded picture.
25. The device of claim 22 , wherein to filter the portion of the decoded picture using the one or more neural network models and the data from the one or more other units of the device, the one or more processors are configured to execute the neural network filtering unit to provide the data from the one or more other units of the device as one or more additional input planes to a convolutional neural network (CNN).
26. The device of claim 22 , wherein to filter the portion of the decoded picture using the one or more neural network models and the data from the one or more other units of the device, the one or more processors are configured to execute the neural network filtering unit to adjust output of the one or more neural network models using the data from the one or more other units of the device.
27. The device of claim 22 , wherein to filter the portion of the decoded picture using the one or more neural network models and the data from the one or more other units of the device, the one or more processors are configured to execute the neural network filtering unit to adjust the data from the one or more other units of the device prior to filtering the portion of the decoded picture.
28. The device of claim 22 , wherein to filter the portion of the decoded picture using the one or more neural network models and the data from the one or more other units of the device, the one or more processors are configured to execute the neural network filtering unit to:
convert deblocking filter data for the decoded picture from the deblocking unit to one or more input planes for at least one of the one or more neural network models; and
filter the portion of the decoded picture using the one or more input planes as inputs to the at least one of the one or more neural network models.
29. The device of claim 22 , further comprising a display configured to display the decoded picture of the video data.
30. The device of claim 22 , wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.
31. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause a processor of a video decoding device to execute a neural network filtering unit to:
receive data for a decoded picture of video data;
receive data from one or more other units of the video decoding device, the data from the one or more other units of the video decoding device being different than the data for the decoded picture, and wherein the instructions that cause the processor to receive the data from the one or more other units of the video decoding device comprise instructions that cause the processor to receive boundary strength data from a deblocking unit of the video decoding device;
determine one or more neural network models to be used to filter a portion of the decoded picture; and
filter the portion of the decoded picture using the one or more neural network models and the data from the one or more other units of the video decoding device, including the boundary strength data.
32. A device for filtering decoded video data, the device comprising a filtering unit comprising:
means for receiving data for a decoded picture of video data;
means for receiving data from one or more other units of the video decoding device, the data from the one or more other units being different than the data for the decoded picture, and wherein the means for receiving the data from the one or more other units of the video decoding device comprises means for receiving boundary strength data from a deblocking unit of the video decoding device;
means for determining one or more neural network models to be used to filter a portion of the decoded picture; and
means for filtering the portion of the decoded picture using the one or more neural network models and the data from the one or more other units of the video decoding device, including the boundary strength data.