On neural network-based filtering for imaging/video coding
A method of processing video data. The method includes selecting an in-loop filter from a plurality of neural network (NN) filter model candidates, wherein the plurality of NN filter model candidates are based on a reconstructed quality level of a video unit, and performing a conversion between a video media file comprising the video unit and a bitstream based on the in-loop filter selected. A corresponding video coding apparatus and non-transitory computer readable medium are also disclosed.
1. A method of processing video data, comprising:
selecting an in-loop filter from a plurality of neural network (NN) filter model candidates, wherein the plurality of NN filter model candidates are based on a reconstructed quality level of a video unit; and
performing a conversion between a video media file comprising the video unit and a bitstream based on the in-loop filter selected,
wherein the method further comprises determining whether a group of NN filter model candidates are the same or different for video units across different temporal layers,
wherein syntax elements corresponding to the in-loop filter selected are coded in the bitstream before syntax elements corresponding to an adaptive loop filter (ALF), and
wherein the method further comprises determining that a same index in a bitstream is associated with different NN filters for two video units.
2. The method of claim 1 , wherein the one or more NN filter model candidates comprise one or more pretrained convolutional neural network (CNN) filter models.
3. The method of claim 1 , wherein each of the plurality of NN filter model candidates corresponds to a different reconstructed quality level of video unit.
4. The method of claim 1 , wherein the reconstructed quality level of the video unit corresponds to a quantization parameter (QP) of the video unit or at least one of a constant rate factor and a bitrate of the video unit.
5. The method of claim 1 , wherein:
the in-loop filter selected is one of a plurality of in-loop filters including a second in-loop filter, and wherein application of the second in-loop filter is dependent on whether or how the in-loop filter selected is applied; and
the plurality of in-loop filters comprise at least one of a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a cross-component adaptive loop filter (CCALF), and a bilateral filter.
6. The method of claim 1 , wherein syntax elements corresponding to the in-loop filter selected are coded in the bitstream at a coding tree unit (CTU) level before syntax elements corresponding to an adaptive loop filter (ALF) or before syntax elements corresponding to a sample adaptive offset (SAO) filter.
7. The method of claim 1 , wherein the in-loop filter selected is coded in a supplemental enhancement information (SEI) message of the bitstream.
8. The method of claim 1 , wherein the same index is disposed in a supplemental enhancement information (SEI) message in the bitstream.
9. The method of claim 1 , wherein the determination of whether the group of NN filter model candidates are the same or different for video units across different temporal layers is specified in a rule included in a supplemental enhancement information (SEI) message of the bitstream.
10. The method of claim 1 , wherein a rule included in the bitstream specifies that a first subgroup of the group of NN filter model candidates is to be used in an in-loop filtering operation across a first subgroup of the different temporal layers, and that a second subgroup of the group of NN filter model candidates is to be used in an in-loop filtering operation across a second subgroup of the different temporal layers.
11. The method of claim 10 , wherein the first subgroup of the different temporal layers comprises layers having a temporal index of no greater than K1, and wherein at least one of the one or more NN filter model candidates to be used in the in-loop filtering operation across the first subgroup is specified by a rule included in the bitstream based on a number of intra coded samples of the first subgroup.
12. The method of claim 10 , wherein a rule included in the bitstream associates the group of NN filter model candidates with both a first temporal layer and a separate second temporal layer of the different temporal layers.
13. The method of claim 10 , wherein a rule included in the bitstream associates the group of NN filter model candidates with a specific temporal layer of the different temporal layers.
14. The method of claim 1 , wherein the conversion includes encoding the video media file into the bitstream.
15. The method of claim 1 , wherein the conversion includes decoding the video media file from the bitstream.
16. An apparatus for coding video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor cause the processor to:
select an in-loop filter from a plurality of neural network (NN) filter model candidates, wherein the plurality of NN filter model candidates are based on a reconstructed quality level of a video unit; and
convert between a video media file comprising the video unit and a bitstream based on the in-loop filter selected,
wherein the instructions, upon execution by the processor, further cause the processor to determine whether a group of NN filter model candidates are the same or different for video units across different temporal layers,
wherein syntax elements corresponding to the in-loop filter selected are coded in the bitstream before syntax elements corresponding to an adaptive loop filter (ALF), and
wherein the processor is further caused to determine that a same index in a bitstream is associated with different neural network (NN) filters for two video units.
17. A method for storing a bitstream of a video, comprising:
selecting an in-loop filter from a plurality of neural network (NN) filter model candidates, wherein the plurality of NN filter model candidates are based on a reconstructed quality level of a video unit;
performing a conversion between a video media file comprising the video unit and a bitstream based on the in-loop filter selected; and
storing the bitstream in a non-transitory computer-readable recording medium,
wherein the method further comprises determining whether a group of NN filter model candidates are the same or different for video units across different temporal layers,
wherein syntax elements corresponding to the in-loop filter selected are coded in the bitstream before syntax elements corresponding to an adaptive loop filter (ALF), and
wherein the method further comprises determining that a same index in a bitstream is associated with different neural network (NN) filters for two video units.