Super resolution position and network structure

1 . A method of processing video data, comprising:

applying, during a conversion between a video comprising a video unit and a bitstream of the video, a super resolution (SR) process to the video unit at a specific position relative to one or more in-loop filters when the one or more in-loop filters are applied to the video unit; and

performing the conversion based on the SR process and the one or more in-loop filters as applied;

wherein the specific position is immediately after the video unit has been reconstructed;

wherein the SR process is applied by a neural network (NN)-based SR network,

wherein an input of the SR process comprises a luma component, and wherein an output of the SR process comprises an up-sampled chroma component, or alternatively,

wherein the input of the SR process comprises a chroma component, and wherein the output of the SR process comprises an up-sampled luma component, and

wherein an up-sampling layer configured to up-sample a resolution of the video unit applies de-convolution with a stride of K to up-sample the resolution of the video unit, where K is an integer greater than 1, and wherein K is dependent on decoded information.

2 . The method of claim 1 , wherein the specific position of the SR process is prior to the one or more in-loop filters.

3 . The method of claim 1 , wherein the video unit is a block, and wherein the block comprises one of a coding tree unit (CTU) or a coding tree block (CTB), or wherein the video unit is a region, and wherein the region comprises a CTU row.

4 . The method of claim 1 , the specific position satisfies at least one of the following conditions:

wherein the one or more in-loop filters comprise a deblocking filter (DBF), and wherein the specific position is prior to the DBF,

wherein the one or more in-loop filters comprise the DBF and a sample adaptive offset (SAO), and wherein the specific position is after the DBF and prior to the SAO,

wherein the one or more in-loop filters comprise the SAO and an adaptive loop filter (ALF), and wherein the specific position is after the SAO and prior to the ALF,

wherein the one or more in-loop filters comprise the ALF, and wherein the specific position is after the ALF, and

wherein the one or more in-loop filters comprise the SAO, and wherein the specific position is prior to the SAO, or wherein the one or more in-loop filters comprise the ALF, and wherein the specific position is prior to the ALF.

5 . The method of claim 1 , wherein the one or more in-loop filters comprise a first filter, and whether the specific position is prior to the first filter depends on whether the first filter uses information from an original image.

6 . The method of claim 1 , wherein an indication of the specific position is included in the bitstream, or wherein the specific position is derived based on decoded information.

7 . The method of claim 1 , further comprising applying the SR process to the video unit without regard to the specific position when the one or more in-loop filters are not applied to the video unit,

wherein the one or more in-loop filters comprise a deblocking filter (DBF), a sample adaptive offset (SAO) and an adaptive loop filter (ALF), and wherein the method further comprises applying the SR process to the video unit without regard to the specific position when the DBF, the SAO, and the ALF are disabled, or

wherein the one or more in-loop filters comprise the ALF, and wherein the method further comprises applying the SR process to the video unit without regard to the specific position when the ALF is disabled, or

wherein the one or more in-loop filters comprise a cross component adaptive loop filter (CC-ALF), wherein the video unit comprises chroma components, and wherein the method further comprises applying the SR process to the chroma components without regard to the specific position when the CC-ALF is disabled.

8 . The method of claim 7 , wherein signaling of side information for the one or more in-loop filters is dependent on application of the SR process, and wherein application of the SR process is dependent on a usage of an in-loop filtering method.

9 . The method of claim 1 , wherein the NN-based SR network comprises a plurality of convolutional layers, and wherein the plurality of convolutional layers includes the up-sampling layer configured to up-sample the resolution of the video unit.

10 . The method of claim 9 , wherein the up-sampling layer applies pixel shuffling to up-sample the resolution of the video unit,

wherein a resolution of an input of the NN-based SR network is 1/K of an original input, where K is a down-sampling ratio,

wherein a first convolutional layer from the plurality of convolutional layers is configured to fuse the input of the NN-based SR network to generate feature maps,

wherein the NN-based SR network further comprises sequentially stacked residual blocks, and wherein the sequentially stacked residual blocks are configured to process the feature maps received from the first convolutional layer,

wherein a last convolutional layer from the plurality of convolutional layers is configured to process feature maps received from a last residual block from the sequentially stacked residual blocks to produce R feature maps, where R equals K*K, and

wherein a shuffle layer is configured to generate the video unit based on the R feature maps, and wherein a spatial resolution of the video unit is the same as a spatial resolution of the original input.

11 . The method of claim 10 , wherein one or more of the sequentially stacked residual blocks consists of, in sequential order, the first convolutional layer, a parametric rectified linear unit (PReLU) activation function, and a second convolutional layer from the plurality of convolutional layers, and wherein an input of the first convolutional layer is added to an output of the second convolutional layer.

12 . The method of claim 1 , wherein the input of the SR process is at a video unit level, wherein the video unit is a sequence of pictures, a picture, a slice, a tile, a brick, a subpicture, one or more coding tree units (CTUs), a CTU row, one or more coding units (CUs), one or more coding tree blocks (CTBs), or a region covering more than one pixel.

13 . The method of claim 1 , wherein the input of the SR process satisfies one of:

wherein the input of the SR process is a coding tree unit (CTU), and wherein the CTU has been down-sampled,

wherein the input of the SR process is a frame, and wherein the frame has been down-sampled, or

wherein the input of the SR process comprises at least one of: a luma component of reconstruction, a chroma component of reconstruction, or both luma and chroma components of a same reconstruction.

14 . The method of claim 1 , wherein the input of the SR process comprises decoded side information, and wherein the input of the SR process comprises a prediction picture, and wherein the output of the SR process is an up-sampled prediction picture.

15 . The method of claim 1 , wherein the conversion includes encoding the video into the bitstream.

16 . The method of claim 1 , wherein the conversion includes decoding the video from the bitstream.

17 . An apparatus for processing media data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to:

apply, during a conversion between a video comprising a video unit and a bitstream of the video, a super resolution (SR) process to the video unit at a specific position relative to one or more in-loop filters when the one or more in-loop filters are applied to the video unit; and

perform the conversion based on the SR process and the one or more in-loop filters as applied;

wherein the specific position is immediately after the video unit has been reconstructed;

wherein the SR process is applied by a neural network (NN)-based SR network,

wherein an input of the SR process comprises a luma component, and wherein an output of the SR process comprises an up-sampled chroma component, or alternatively,

wherein the input of the SR process comprises a chroma component, and wherein the output of the SR process comprises an up-sampled luma component, and

wherein an up-sampling layer configured to up-sample a resolution of the video unit applies de-convolution with a stride of K to up-sample the resolution of the video unit, where K is an integer greater than 1, and wherein K is dependent on decoded information.

18 . A method for storing a bitstream of a video, comprising:

applying, for the video comprising a video unit, a super resolution (SR) process to the video unit at a specific position relative to one or more in-loop filters when the one or more in-loop filters are applied to the video unit;

generating the bitstream based on the SR process and the one or more in-loop filters as applied; and

storing the bitstream in a non-transitory computer-readable recording medium,

wherein the specific position is immediately after the video unit has been reconstructed;

wherein the SR process is applied by a neural network (NN)-based SR network,

wherein an input of the SR process comprises a luma component, and wherein an output of the SR process comprises an up-sampled chroma component, or alternatively,

wherein the input of the SR process comprises a chroma component, and wherein the output of the SR process comprises an up-sampled luma component, and

wherein an up-sampling layer configured to up-sample a resolution of the video unit applies de-convolution with a stride of K to up-sample the resolution of the video unit, where K is an integer greater than 1, and wherein K is dependent on decoded information.