Methods and non-transitory computer readable storage medium for spatial resampling towards machine vision
A method of encoding a video sequence into a bitstream. The method includes receiving a video sequence; performing a plurality of convolutions on an input image data of the video sequence in YUV format; wherein performing the plurality of convolutions includes performing a first stage convolution on the input image data, wherein the first stage convolution comprises a first convolution and a second convolution that are provided in parallel; performing a second stage convolution on a channel-wise concatenation result of an output of the first convolution and an output of the second convolution; performing a third stage convolution on an output of the second stage convolution; and obtaining an output image data based on an output of the third stage convolution; and encoding the output image data for generating the bitstream.
1 . A method of decoding a bitstream to output one or more pictures for a video stream, the method comprising:
receiving a bitstream; and
decoding, using coded information of the bitstream, one or more pictures comprising a down-sampled image data in YUV format; and
performing a plurality of convolutions on the down-sampled image data, wherein performing the plurality of convolutions comprises:
performing a first stage convolution on the down-sampled image data, wherein the first stage convolution comprises a first convolution and a second convolution provided in parallel;
performing a second stage convolution on a channel-wise concatenation result of an output of the first convolution and an output of the second convolution;
performing a third stage convolution on an output of the second stage convolution, wherein the third stage convolution comprises a fourth convolution and a fifth convolution provided in parallel;
performing a bicubic interpolation on the down-sampled image data to obtain a bicubic interpolation result; and
performing an element-wise addition to an output of third stage convolution and the bicubic interpolation result to obtain an up-sampled image data;
wherein performing the element-wise addition to the output of third stage convolution and the bicubic interpolation result to obtain the up-sampled image data, further comprises:
performing a first element-wise addition to an output of the fourth convolution and a first bicubic interpolation result of a Y component of the down-sampled image data to obtain a Y component of the up-sampled image data; and
performing a second element-wise addition to an output of the fifth convolution and a second bicubic interpolation result of a channel-wise concatenation result of a U component and a V component of the down-sampled image data to obtain a channel-wise concatenation result of a U component and a V component of the up-sampled image data.
2 . The method according to claim 1 , wherein performing the first stage convolution on the image data comprises:
performing the first convolution on a Y component of the down-sampled image data; and
performing the second convolution on a channel-wise concatenation result of a U component and a V component of the down-sampled image data.
3 . The method according to claim 1 , wherein the second stage convolution comprises a series of convolutions.
4 . The method according to claim 1 , wherein performing the third stage convolution on the output of the second stage convolution further comprises:
performing the fourth convolution on the output of the second stage convolution; and
performing the fifth convolution on the output of the second stage convolution.
5 . The method according to claim 1 , wherein a set of parameters of each of the plurality of convolutions comprises: an input channel number, an output channel number, a kernel size, a stride, and a padding size.
6 . The method according to claim 5 , wherein the set of parameters is determined based on a type of a YUV format and a number of the plurality of convolutions.
7 . The method according to claim 1 , wherein a Rectified Linear Unit (ReLU) is applied to each convolution in the first stage convolution and the second stage convolution as an activation function.
8 . An apparatus for image data processing, comprising:
a memory configured to store instructions; and
one or more processors configured to execute the instructions to cause the apparatus to perform operations comprising:
performing a plurality of convolutions on a down-sampled image data in YUV format, wherein performing the plurality of convolutions comprises:
performing a first stage convolution on the down-sampled image data, wherein the first stage convolution comprises a first convolution and a second convolution provided in parallel;
performing a second stage convolution on a channel-wise concatenation result of an output of the first convolution and an output of the second convolution;
performing a third stage convolution on an output of the second stage convolution, wherein the third stage convolution comprises a fourth convolution and a fifth convolution provided in parallel;
performing a bicubic interpolation on the down-sampled image data to obtain a bicubic interpolation result; and
performing an element-wise addition to an output of third stage convolution and the bicubic interpolation result to obtain an up-sampled image data;
wherein performing the element-wise addition to the output of third stage convolution and the bicubic interpolation result to obtain the up-sampled image data, further comprises:
performing a first element-wise addition to an output of the fourth convolution and a first bicubic interpolation result of a Y component of the down-sampled image data to obtain a Y component of the up-sampled image data; and
performing a second element-wise addition to an output of the fifth convolution and a second bicubic interpolation result of a channel-wise concatenation result of a U component and a V component of the down-sampled image data to obtain a channel-wise concatenation result of a U component and a V component of the up-sampled image data.
9 . The apparatus according to claim 8 , wherein performing the first stage convolution on the down-sampled image data comprises:
performing the first convolution on a Y component of the down-sampled image data; and
performing the second convolution on a channel-wise concatenation result of a U component and a V component of the down-sampled image data.
10 . The apparatus according to claim 8 , wherein the second stage convolution comprises a series of convolutions.
11 . The apparatus according to claim 8 , wherein performing the third stage convolution on the output of the second stage convolution further comprises:
performing the fourth convolution on the output of the second stage convolution; and
performing the fifth convolution on the output of the second stage convolution.
12 . The apparatus according to claim 8 , wherein a set of parameters of each of the plurality of convolutions comprises: an input channel number, an output channel number, a kernel size, a stride, and a padding size.
13 . The apparatus according to claim 12 , wherein the set of parameters is determined based on a type of a YUV format and a number of the plurality of convolutions.
14 . The apparatus according to claim 8 , wherein a Rectified Linear Unit (ReLU) is applied to each convolution in the first stage convolution and the second stage convolution as an activation function.
15 . A non-transitory computer readable medium that stores a set of instructions that is executable by one or more processors of an apparatus to cause the apparatus to perform operations comprising:
performing a plurality of convolutions on a down-sampled image data in YUV format, wherein performing the plurality of convolutions comprises:
performing a first stage convolution on the down-sampled image data, wherein the first stage convolution comprises a first convolution and a second convolution provided in parallel;
performing a second stage convolution on a channel-wise concatenation result of an output of the first convolution and an output of the second convolution;
performing a third stage convolution on an output of the second stage convolution, wherein the third stage convolution comprises a fourth convolution and a fifth convolution provided in parallel;
performing a bicubic interpolation on the down-sampled image data to obtain a bicubic interpolation result; and
performing an element-wise addition to an output of third stage convolution and the bicubic interpolation result to obtain an up-sampled image data;
wherein performing the element-wise addition to the output of third stage convolution and the bicubic interpolation result to obtain the up-sampled image data, further comprises:
performing a first element-wise addition to an output of the fourth convolution and a first bicubic interpolation result of a Y component of the down-sampled image data to obtain a Y component of the up-sampled image data; and
performing a second element-wise addition to an output of the fifth convolution and a second bicubic interpolation result of a channel-wise concatenation result of a U component and a V component of the down-sampled image data to obtain a channel-wise concatenation result of a U component and a V component of the up-sampled image data.
16 . The non-transitory computer readable medium according to claim 15 , wherein performing the first stage convolution on the down-sampled image data comprises:
performing the first convolution on a Y component of the down-sampled image data; and
performing the second convolution on a channel-wise concatenation result of a U component and a V component of the down-sampled image data.
17 . The non-transitory computer readable medium according to claim 15 , wherein the second stage convolution comprises a series of convolutions.
18 . The non-transitory computer readable medium according to claim 15 , wherein performing the third stage convolution on the output of the second stage convolution further comprises:
performing the fourth convolution on the output of the second stage convolution; and
performing the fifth convolution on the output of the second stage convolution.
19 . The non-transitory computer readable medium according to claim 15 , wherein a set of parameters of each of the plurality of convolutions comprises: an input channel number, an output channel number, a kernel size, a stride, and a padding size.
20 . The non-transitory computer readable medium according to claim 19 , wherein the set of parameters is determined based on a type of a YUV format and a number of the plurality of convolutions.