Content-weighted deep residual learning for video in-loop filtering
Systems and methods are provided for improving filtering performance and Bjøntegaard-Delta (BD) rate savings for video processing. In addition to computing the artifacts between a given compressed image and a restored clean image after filtering using Deep Residual Learning (DRL) for recovering the residual between input and output, filtering strength of a loop filter may be controlled by the content of the region of the image, such that, in more important areas, such as the face and edges, the filtering strength may be increased while in less important areas, such as textures and backgrounds, the filtering strength may be decreased.
1. A method comprising:
receiving an image;
generating a content weight map of the image in a content weight network (CWN);
generating a feature map of the image in a deep residual network (DRN);
generating a residual map of the image based on the content weight map and the feature map; and
combining the residual map with the image to compute an output image.
2. The method of claim 1 , wherein generating the content weight map of the image includes:
dividing the image into a plurality of regions;
computing a weight of each region of the plurality of regions; and
generating the content weight map of the image based on the weight of each region of the plurality of regions.
3. The method of claim 2 , wherein computing the weight of each region of the plurality of regions includes:
determining a corresponding feature of each region of the plurality of regions; and
computing the weight of each region of the plurality of regions based on the corresponding feature.
4. The method of claim 3 , wherein the corresponding feature of each region of the plurality of regions includes one of:
a level of details required for the corresponding region for the output image,
quantization parameters,
motion vectors, and
coding unit partition information.
5. The method of claim 1 , wherein generating the feature map of the image includes:
extracting features from the image by a convolutional layer, and
training the DRN using a rectified linear unit (ReLu) layer.
6. The method of claim 5 , wherein the convolutional layer and the ReLu layer form a first layer pair of a plurality of layer pairs, and an output from a layer pair is used as an input for a subsequent layer pair.
7. The method of claim 1 , wherein generating the residual map of the image based on the content weight map and the feature map include:
elementwise multiplying the content weight map and the feature map to generate a multiplied output, and
inputting the multiplied output into a multiplier convolutional layer to generate the residual map of the image.
8. The method of claim 1 , wherein combining the residual map with the image to compute the output image includes:
elementwise summing the residual map to the image to compute the output image.
9. A loop filtering system comprising:
an image receiver configured to receive an image;
a content weight network (CWN) coupled to the image receiver, the CWN configured to receive the image from the image receiver to generate a content weight map of the image;
a deep residual network (DRN) coupled to the image receiver, the DRN configured to receive the image from the image receiver to generate a feature map of the image;
a multiplier coupled to the CWN and the DRN, the multiplier configured to generate a residual map of the image based on the content weight map and the feature map; and
an adder coupled to the image receiver and the multiplier, the adder configured to receive the image and to combine the residual map with the image to compute an output image.
10. The loop filtering system of claim 9 , wherein the CWN is further configured to generate the content weight map of the image by:
dividing the image into a plurality of regions;
computing a weight of each region of the plurality of regions; and
generating the content weight map of the image based on the weight of each region of the plurality of regions.
11. The loop filtering system of claim 10 , wherein computing the weight of each region of the plurality of regions includes:
determining a corresponding feature of each region of the plurality of regions; and
computing the weight of each region of the plurality of regions based on the corresponding feature.
12. The loop filtering system of claim 11 , wherein the corresponding feature of each region of the plurality of regions includes one of:
a level of details required for the corresponding region for the output image,
quantization parameters,
motion vectors, and
coding unit partition information.
13. The loop filtering system of claim 9 , wherein the DRN comprises a layer pair including:
a convolutional layer configured to extract features from the image, and
a rectified linear unit (ReLu) layer coupled to the convolutional layer, the ReLu layer configured to train the DRN.
14. The loop filtering system of claim 13 , wherein the layer pair is a first layer pair of a plurality of layer pairs, and an output from a layer pair is used as an input for a subsequent layer pair.
15. The loop filtering system of claim 9 , wherein the multiplier comprises a multiplier convolutional layer, the multiplier further configured to:
elementwise multiply the content weight map and the feature map to generate a multiplied output, and
input the multiplied output into the multiplier convolutional layer to generate the residual map of the image.
16. The loop filtering system of claim 9 , wherein the adder is further configured to combine the residual map with the image by elementwise summing the residual map to the image to compute the output image.
17. A computer-readable storage medium storing computer-readable instructions executable by one or more processors, that when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving an image;
generating a content weight map of the image in a content weight network (CWN);
generating a feature map of the image in a deep residual network (DRN);
generating a residual map of the image based on the content weight map and the feature map;
elementwise multiplying the content weight map and the feature map to generate a multiplied output;
inputting the multiplied output into a multiplier convolutional layer to generate the residual map of the image; and
elementwise summing the residual map to the image to compute the output image.
18. The computer-readable storage medium of claim 17 , wherein generating the content weight map of the image includes:
dividing the image into a plurality of regions;
computing a weight of each region of the plurality of regions; and
generating the content weight map of the image based on the weight of each region of the plurality of regions.
19. The computer-readable storage medium of claim 18 , wherein computing the weight of each region of the plurality of regions includes:
determining a corresponding feature of each region of the plurality of regions; and
computing the weight of each region of the plurality of regions based on the corresponding feature.
20. The computer-readable storage medium of claim 19 , wherein the corresponding feature of each region of the plurality of regions includes one of:
a level of details required for the corresponding region for the output image,
quantization parameters,
motion vectors, and
coding unit partition information.