LIGHTWEIGHT SPATIAL UPSAMPLING METHODS FOR MACHINE VISION
The present disclosure provides spatial upsampling models used for processing video data suitable for machine vision tasks. An exemplary decoding method includes: receiving a bitstream; and decoding, using coded information of the bitstream, one or more pictures, wherein the decoding includes: generating one or more decompressed pictures by decompressing one or more compressed pictures included in the bitstream; and performing spatial upsampling on the one or more decompressed pictures by a spatial upsampling model to obtain one or more reconstructed pictures, respectively, wherein a total length of coding bits of parameters of the spatial upsampling model is less than a threshold that is pre-determined based on a desired quality of the reconstructed pictures.
1 . A method for decoding a bitstream to output one or more pictures for a video stream, the method comprising:
receiving a bitstream; and
decoding, using coded information of the bitstream, one or more pictures,
wherein the decoding comprises:
decompressing one or more compressed pictures comprised in the bitstream; and
performing spatial upsampling on the one or more decompressed pictures by a spatial upsampling model to obtain one or more reconstructed pictures, respectively,
wherein a total length of coding bits of parameters of the spatial upsampling model is less than a threshold that is pre-determined based on a desired quality of the reconstructed pictures.
2 . The method of claim 1 , wherein the decoding further comprises:
extracting from the bitstream parameters that comprise weights and bias for respective layers of the spatial upsampling model.
3 . The method of claim 1 , wherein the spatial upsampling model comprises 11 convolutional layers.
4 . The method of claim 3 , wherein the spatial upsampling model comprises:
an input convolutional layer having a kernel size being 3, a stride being 1, a number of input channels being 3, and a number of output channels being 64;
nine middle convolutional layers having a kernel size being 3, a stride being 1, a number of input channels being 64, and a number of output channels being 64; and
an output convolutional layer having a kernel size being 3, a stride being 1, a number of input channels being 64, and a number of output channels being 3.
5 . The method of claim 1 , wherein the spatial upsampling model comprises one or more Bottleneck Resblocks (BRes).
6 . The method of claim 5 , wherein the spatial upsampling model comprises:
an input convolutional layer having a kernel size being 3, a stride being 1, a number of input channels being 3, and a number of output channels being 64;
an output convolutional layer having a kernel size being 3, a stride being 1, a number of input channels being 64, and a number of output channels being 3; and
three BRes arranged between the input convolutional layer and the output convolutional layer.
7 . The method of claim 6 , wherein the BRes comprises:
a first convolutional layer having a kernel size being 1, a stride being 1, a number of input channels being 64, and a number of output channels being 32;
a second convolutional layer having a kernel size being 3, a stride being 1, a number of input channels being 32, and a number of output channels being 32; and
a third convolutional layer having a kernel size being 1, a stride being 1, a number of input channels being 32, and a number of output channels being 64.
8 . The method of claim 1 , wherein the parameters of the spatial upsampling model are quantized into a pre-determined format.
9 . The method of claim 8 , wherein the pre-determined format is one of float16, int8, or binary.
10 . A method for encoding a video sequence into a bitstream, the method comprising:
receiving a video sequence;
encoding one or more pictures of the video sequence; and
generating a bitstream associated with the one or more pictures,
wherein the encoding comprises:
performing spatial downsampling on the one or more pictures by a spatial downsampling model to obtain one or more downsampled pictures, respectively;
compressing the one or more downsampled pictures; and
generating parameters for a spatial upsampling model for decoding the one or more compressed pictures,
wherein a total length of coding bits of the parameters of the spatial upsampling model is less than a threshold that is pre-determined based on a desired quality of reconstructed pictures of the one or more pictures.
11 . The method of claim 10 , wherein generating the bitstream associated with the encoded one or more pictures comprises:
signaling the parameters of the spatial upsampling model into the bitstream, the parameters comprising weights and bias for respective layers of the spatial upsampling model.
12 . The method of claim 10 , wherein the spatial upsampling model comprises 11 convolutional layers.
13 . The method of claim 12 , wherein the spatial upsampling model comprises:
an input convolutional layer having a kernel size being 3, a stride being 1, a number of input channels being 3, and a number of output channels being 64;
nine middle convolutional layers having a kernel size being 3, a stride being 1, a number of input channels being 64, and a number of output channels being 64; and
an output convolutional layer having a kernel size being 3, a stride being 1, a number of input channels being 64, and a number of output channels being 3.
14 . The method of claim 10 , wherein the spatial upsampling model comprises one or more Bottleneck Resblocks (BRes).
15 . The method of claim 14 , wherein the spatial upsampling model comprises:
an input convolutional layer having a kernel size being 3, a stride being 1, a number of input channels being 3, and a number of output channels being 64;
an output convolutional layer having a kernel size being 3, a stride being 1, a number of input channels being 64, and a number of output channels being 3; and
three BRes arranged between the input convolutional layer and the output convolutional layer.
16 . The method of claim 15 , wherein the BRes comprises:
a first convolutional layer having a kernel size being 1, a stride being 1, a number of input channels being 64, and a number of output channels being 32;
a second convolutional layer having a kernel size being 3, a stride being 1, a number of input channels being 32, and a number of output channels being 32; and
a third convolutional layer having a kernel size being 1, a stride being 1, a number of input channels being 32, and a number of output channels being 64.
17 . The method of claim 10 , wherein the parameters of the spatial upsampling model are quantized into a pre-determined format.
18 . The method of claim 17 , wherein the pre-determined format is one of float16, int8, or binary.
19 . A non-transitory computer readable storage medium storing a bitstream of a video for processing according to operations comprising:
generating one or more decompressed pictures by decompressing one or more compressed pictures comprised in the bitstream; and
performing spatial upsampling on the one or more decompressed pictures by a spatial upsampling model to obtain one or more reconstructed pictures, respectively,
wherein a total length of coding bits of parameters of the spatial upsampling model is less than a threshold that is pre-determined based on a desired quality of the reconstructed pictures.
20 . The non-transitory computer readable storage medium of claim 19 , wherein the spatial upsampling model comprises 11 convolutional layers, or one or more Bottleneck Resblocks (BRes).