Using neural network filtering in video coding
View Patent ↗Methods, systems, apparatus for media processing are described. One example method of digital media processing includes determining, for a conversion between visual media data and a bitstream of the visual media data, how to apply one or more convolutional neural network filters to at least some samples of a video unit of the visual media data according to a rule; and performing the conversion based on the determining.
1 . A method of processing visual media data, comprising:
determining, for a conversion between visual media data and a bitstream of the visual media data, how to apply one or more convolutional neural network filters to samples of a video unit of the visual media data according to a rule; and
performing the conversion based on the determining,
wherein when the one or more convolutional neural network filters are applied, a difference between a convolutional neural network filtered sample and its unfiltered version is clipped to a range,
wherein the rule specifies that a selection of a set of convolutional neural network filters depends on a group of pictures (GOP) size of the video unit, and
wherein the video unit is a slice or a picture or a tile or a subpicture or a coding tree block or a coding tree unit.
2 . The method of claim 1 , wherein a convolutional neural network filer is implemented using a convolutional neural network.
3 . The method of claim 1 , wherein the rule specifies that the determining is based on decoded information associated with a video unit of the visual media data, wherein the decoded information includes at least one of prediction modes, transform types, a skip flag, or coded block flag (CBF) values.
4 . The method of claim 1 , wherein the rule specifies that information related to the one or more convolutional neural network filters is controlled at a granularity smaller than a video unit of the visual media data.
5 . The method of claim 4 , wherein the information is controlled at a sample or pixel level.
6 . The method of claim 4 , wherein the information is controlled at a sample row, a sample column, or a sample line level.
7 . The method of claim 4 , wherein the rule specifies that a set of convolutional neural network filters is determined based on a value of a sample or a position of the sample within the video unit of the visual media data.
8 . The method of claim 1 , wherein the rule specifies that a selection of a set of convolutional neural network filters depends on a temporal layer identification of a video unit of the visual media data.
9 . The method of claim 8 , wherein the temporal layer identification is classified to more than one category and a given set of convolutional neural network filters is applied for a corresponding category.
10 . The method of claim 9 , wherein classification of the temporal layer identification is based on the GOP size.
11 . The method of claim 1 , wherein the rule specifies that a set of convolutional neural network filters is utilized for video units with different temporal layers.
12 . The method of claim 1 , wherein the performing of the conversion comprises generating the bitstream from the visual media data.
13 . The method of claim 1 , wherein the performing of the conversion comprises generating the visual media data from the bitstream.
14 . An apparatus for processing visual media data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to:
determine, for a conversion between visual media data and a bitstream of the visual media data, how to apply one or more convolutional neural network filters to samples of a video unit of the visual media data according to a rule; and
perform the conversion based on the determining,
wherein when the one or more convolutional neural network filters are applied, a difference between a convolutional neural network filtered sample and its unfiltered version is clipped to a range,
wherein the rule specifies that a selection of a set of convolutional neural network filters depends on a group of pictures (GOP) size of the video unit, and
wherein the video unit is a slice or a picture or a tile or a subpicture or a coding tree block or a coding tree unit.
15 . The apparatus of claim 14 , wherein the rule specifies that the determining is based on decoded information associated with a video unit of the visual media data, wherein the decoded information includes at least one of prediction modes, transform types, a skip flag, or coded block flag (CBF) values.
16 . The apparatus of claim 14 , wherein the rule specifies that information related to the one or more convolutional neural network filters is controlled at a granularity smaller than a video unit of the visual media data.
17 . A non-transitory computer-readable storage medium storing instructions that cause a processor to:
determine, for a conversion between visual media data and a bitstream of the visual media data, how to apply one or more convolutional neural network filters to samples of a video unit of the visual media data according to a rule; and
perform the conversion based on the determining,
wherein when the one or more convolutional neural network filters are applied, a difference between a convolutional neural network filtered sample and its unfiltered version is clipped to a range,
wherein the rule specifies that a selection of a set of convolutional neural network filters depends on a group of pictures (GOP) size of the video unit, and
wherein the video unit is a slice or a picture or a tile or a subpicture or a coding tree block or a coding tree unit.
18 . The non-transitory computer-readable storage medium of claim 17 , wherein the rule specifies that the determining is based on decoded information associated with a video unit of the visual media data, wherein the decoded information includes at least one of prediction modes, transform types, a skip flag, or coded block flag (CBF) values.
19 . The non-transitory computer-readable storage medium of claim 17 , wherein the rule specifies that information related to the one or more convolutional neural network filters is controlled at a granularity smaller than a video unit of the visual media data.
20 . The apparatus of claim 14 , wherein the rule specifies that a selection of a set of convolutional neural network filters depends on a temporal layer identification of a video unit of the visual media data.