Encoding and decoding methods and devices including CNN-based in-loop filter
View Patent ↗Disclosed are an encoding device and a decoding device, which include a CNN-based in-loop filter. The encoding device according to one embodiment comprises: a filtering unit for generating filtering information by filtering a residual image corresponding to a difference between an original image and a prediction image; an inverse filtering unit for generating inverse filtering information by inversely filtering the filtering information; a prediction unit for generating the prediction image on the basis of the original image and reconstruction information; a CNN-based in-loop filter for receiving the inverse filtering information and the prediction image so as to output the reconstruction information; and an encoding unit for performing encoding on the basis of the filtering information and information of the prediction image.
1. A convolutional neural network (CNN)-based in-loop filter learning method comprising:
using a first adder which receives an original image and a prediction image to generate a residual image;
generating filtering information by filtering the residual image corresponding to a difference between an original image and the prediction image based on an artefact value section of a plurality of artefact value sections or a plurality of quantization parameter sections;
generating inverse filtering information by inversely filtering the filtering information;
using a second adder which receives the inverse filtering information and the prediction image to generate a preliminary reconstruction information;
receiving, by an in-loop filter, the preliminary reconstruction information to generate a secondary preliminary reconstruction information by filtering the preliminary reconstruction information;
generating reconstruction information by applying a CNN-based in-loop filter to the secondary preliminary reconstruction information by performing training for the plurality of artefact sections or quantization parameter sections; and
generating the prediction image based on the reconstruction information,
wherein the weight parameters of the CNN-based in-loop filter are associated with a plurality of connection weight parameters between layers of the CNN-based in-loop filter as the weight parameters are determined to minimize one or more predefined loss functions during training, the weight parameters are obtained from at least one of quantization parameter section, an artefact value section, a texture complexity section, a motion complexity section, a slice type, a temporal encoding hierarchy level, an encoding mode, and a neighboring encoding block, and
wherein performing training for the plurality of artefact sections or the plurality of quantization sections comprising:
performing training for each of the plurality of artefact sections or quantization parameter sections by adjusting one of the plurality of connection weight parameters by filtering a reconstruction of the residual image to generate an output image to minimize a difference between a reconstruction of the residual image and the residual image.
2. The method of claim 1 , wherein the generating of the filtering information comprises generating filtering information by transforming and quantizing the residual image, and
the generating of the inverse filtering information comprises generating inverse filtering information by performing inverse quantization and inverse transformation on the filtering information.
3. The method of claim 1 , wherein the generating of the filtering information comprises filtering the residual image based on a quantization section by a quantization parameter.
4. The method of claim 1 , wherein the generating of the filtering information comprises filtering the residual image based on an artefact value section by an artefact value.
5. The method of claim 1 , wherein the generating of the filtering information comprises filtering the residual image based on a texture complexity section of an image characteristic.
6. The method of claim 1 , wherein the generating of the filtering information comprises filtering the residual image based on a motion complexity section of an image characteristic.
7. The method of claim 1 , wherein the generating of the reconstruction information comprises generating reconstruction information by inputting the inverse filtering information and prediction information based on the prediction image to the CNN-based in-loop filter, and
the reconstruction information is in the same format as that of the residual image.
8. The method of claim 7 , wherein the generating of the reconstruction information by inputting the inverse filtering information and prediction information based on the prediction image to the CNN-based in-loop filter comprises performing in-loop filtering on the prediction information.
9. An encoding apparatus comprising:
a first adder configured to receive an original image and a prediction image to generate a residual image;
a filtering unit configured to generate filtering information by filtering the residual image corresponding to a difference between an original image and the prediction image based on an artefact value section of a plurality of artefact value sections or quantization parameter sections;
an inverse filtering unit configured to generate inverse filtering information by inversely filtering the filtering information;
a second adder configured to receive the inverse filtering information and the prediction image to generate a preliminary reconstruction information;
an in-loop filter configured to receive the preliminary reconstruction information to generate a secondary preliminary reconstruction information by filtering the preliminary reconstruction information;
a convolutional neural network (CNN)-based in-loop filter configured to receive the secondary preliminary reconstruction information to output the reconstruction information by performing training for the plurality of artefact sections or quantization parameter sections; and
an encoder configured to perform encoding based on the filtering information and information of the prediction image,
wherein weight parameters of the CNN-based in-loop filter are associated with a plurality of connection weight parameters between layers of the CNN-based in-loop filter as the weight parameters are determined to minimize one or more loss functions during training, the weight parameters are obtained from at least one of a quantization parameter section, an artefact value section, a texture complexity section, a motion complexity section, a slice type, a temporal encoding hierarchy level, a encoding mode, and a neighboring encoding block, and performing training for the plurality of artefact sections or the plurality of quantization parameter sections comprising:
performing training for each of the plurality of artefact sections or quantization parameter sections by adjusting one of the plurality of connection weight parameters by filtering a reconstruction of the residual image to generate an output image to minimize a difference between a reconstruction of the residual image and the residual image.
10. The encoding apparatus of claim 9 , wherein the filtering unit is configured to generate filtering information by transforming and quantizing the residual image, and
the inverse filtering unit is configured to generate inverse filtering information by performing inverse quantization and inverse transformation on the filtering information.
11. The encoding apparatus of claim 9 , wherein the reconstruction information is in the same format as that of the original image, and
the CNN-based in-loop filter is configured to generate reconstruction information by inputting the inverse filtering information and prediction information based on the prediction image to the CNN-based in-loop filter.
12. The encoding apparatus of claim 11 , further comprising:
an in-loop filter configured to perform in-loop filtering on the prediction information.
13. The encoding apparatus of claim 12 , wherein the in-loop filter comprises at least one of a deblocking filter, a sample adaptive offset filter, and an adaptive loop filter.
14. The encoding apparatus of claim 9 , wherein the reconstruction information is in the same format as that of the residual image, and
the CNN-based in-loop filter is configured to generate reconstruction information by inputting the inverse filtering information and prediction information based on the prediction image to the CNN-based in-loop filter.
15. The encoding apparatus of claim 14 ,
wherein the in-loop filter is configured to generate the secondary preliminary reconstruction information comprising performing in-loop filtering on the prediction information.
16. The encoding apparatus of claim 9 , wherein the
an in-loop filter is configured to generate the secondary preliminary reconstruction information comprising performing in-loop filtering on the reconstruction information.
17. A decoding apparatus comprising:
an entropy decoder configured to output filtering information and preliminary prediction information by decoding encoded bitstream information;
an inverse filtering unit configured to generate inverse filtering information by inversely filtering the filtering information;
an estimator configured to generate a prediction image based on the preliminary prediction information;
a first adder configured to receive the prediction image and the inverse filtering information to generate preliminary reconstruction information;
an in-loop filter configured to receive the preliminary reconstruction information to generate a second preliminary reconstruction information by filtering the preliminary reconstruction information;
a convolutional neural network (CNN)-based in-loop filter configured to generate reconstruction information based on the second preliminary reconstruction information by performing training for the plurality of artefact sections or quantization parameter sections,
wherein the weight parameters of the CNN-based in-loop filter are associated with a plurality of connection weight parameters between layers of the CNN-based in-loop filter as the weight parameters are determined to minimize one or more predefined loss functions during training, the weight parameters are obtained from at least one of a quantization parameter section, an artefact value section, a texture complexity section, a motion complexity section, a slice type, a temporal encoding hierarchy level, a encoding mode, and a neighboring encoding block, and performing training for the plurality of artefact sections comprising:
performing training for each of the plurality of artefact sections or quantization parameter sections by adjusting one of the plurality of connection weight parameters by filtering a reconstruction of the residual image to generate an output image to minimize a difference between a reconstruction of the residual image and the residual image.
18. The decoding apparatus of claim 17 , wherein the reconstruction information is in the same format as that of the original image, and
the CNN-based in-loop filter is configured to generate reconstruction information by inputting the inverse filtering information and prediction information based on the prediction image to the CNN-based in-loop filter.
19. The decoding apparatus of claim 18 , wherein the in-loop filter is configured to generate the secondary reconstruction information comprising performing in-loop filtering on the inverse filtering information.
20. The decoding apparatus of claim 19 , wherein the in-loop filter comprises at least one of a deblocking filter, a sampled adaptive offset filter, and an adaptive loop filter.
21. The decoding apparatus of claim 17 , wherein the reconstruction information is in the same format as that of the residual image, and
the CNN-based in-loop filter is configured to generate reconstruction information by inputting the inverse filtering information and prediction information based on the prediction image to the CNN-based in-loop filter.
22. The decoding apparatus of claim 17 , wherein the in-loop filter comprises at least one of a deblocking filter, a sample adaptive offset filter, and an adaptive loop filter.
23. The decoding apparatus of claim 17 , wherein the reconstruction information is in the same format as that of the residual image, and
the CNN-based in-loop filter is configured to generate residual reconstruction information by inputting the inverse filtering information to the CNN-based in-loop filter.
24. The decoding apparatus of claim 17 , wherein the in loop filter is configured to generate the secondary reconstruction information comprising performing in-loop filtering on the reconstruction information.
25. The decoding apparatus of claim 24 , wherein the in-loop filter comprises at least one of a deblocking filter, a sample adaptive offset filter, and an adaptive loop filter.