End-to-end optimization of adaptive spatial resampling towards machine vision
A computer-implemented method for training spatial resampling modules includes: down-sampling, by a down-sampling module, an input image data to generate a down-sampled image data; up-sampling, by an up-sampling module, the down-sampled image data to generate a first up-sampled image data; analyzing, by a plurality of analysis models corresponding to a plurality of tasks, the first up-sampled image data; and training the down-sampling module based on a loss function associated with the plurality of analysis models according to the input image data and the first up-sampled image data.
1 . A computer-implemented method for spatial resampling, comprising:
performing an instance segmentation to an image to be analyzed;
selecting a resampling factor from a plurality of resampling factor candidates based on an area of object regions calculated according to the instance segmentation;
down-sampling, by a down-sampling module, the image to be analyzed based on the selected resampling factor for resampling the image to generate a down-sampled image data; and
compressing, by an encoder, the down-sampled image data to obtain a quantized and compressed bitstream, wherein the bitstream is decoded by a decoder to obtain a reconstructed image data that is up-sampled, by an up-sampling module, based on the selected resampling factor to generate an up-sampled image data.
2 . The computer-implemented method of claim 1 , further comprising:
up-sampling, by an up-sampling module, the down-sampled image data based on the selected resampling factor to generate an up-sampled image data.
3 . The computer-implemented method of claim 1 , wherein the selecting the resampling factor comprises:
selecting the resampling factor based on a width and a height of the image, and the area of object regions calculated by an instance segmentation network performing the instance segmentation.
4 . The computer-implemented method of claim 1 , further comprising:
skipping the down-sampling in response to the resampling factor being 100 percent when a portion parameter calculated based on the area of object regions is lower than or equal to a threshold value.
5 . The computer-implemented method of claim 1 , wherein the down-sampling module is trained based on a loss function associated with a plurality of analysis models.
6 . A computer-implemented method for spatial resampling, comprising:
decoding, by a decoder, a quantized and compressed bitstream to obtain reconstructed image data and a resampling factor for resampling an image; and
up-sampling, by an up-sampling module, the reconstructed image data based on the resampling factor to generate an up-sampled image data, wherein the resampling factor is selected from a plurality of resampling factor candidates based on an area of object regions calculated according to instance segmentation of the image, wherein the bitstream is obtained by compressing, by an encoder, a down-sampled image data.
7 . The computer-implemented method of claim 6 , wherein the resampling factor is selected based on a width and a height of the image, and the area of object regions calculated by an instance segmentation network performing the instance segmentation.
8 . The computer-implemented method of claim 6 , further comprising:
skipping the up-sampling in response to the resampling factor being 100 percent when a portion parameter calculated based on the area of object regions is lower than or equal to a threshold value.
9 . The computer-implemented method of claim 6 , wherein the up-sampling module is trained based on a loss function associated with a plurality of analysis models.
10 . A method for processing a bitstream, comprising:
receiving a bitstream comprising coded data associated with an input image, wherein the bitstream is obtained by compressing, by an encoder, a down-sampled image data;
decoding, by a decoder, the bitstream to obtain reconstructed image data; and
up-sampling, by an up-sampling module, the reconstructed image data based on a resampling factor to generate an up-sampled image data, wherein the resampling factor is selected from a plurality of resampling factor candidates based on an area of object regions calculated according to instance segmentation of the image.
11 . The method of claim 10 , wherein the bitstream comprises an index representing the resampling factor.
12 . The method of claim 10 , wherein the resampling factor is selected based on a width and a height of the input image, and an area of object regions calculated by an instance segmentation network performing the instance segmentation.
13 . The method of claim 10 , further comprising:
skipping the up-sampling in response to the resampling factor being 100 percent when a portion parameter calculated based on the area of object regions is lower than or equal to a threshold value.
14 . The method of claim 10 , wherein the up-sampling module is trained based on a loss function associated with a plurality of analysis models during a training stage.
15 . The method of claim 14 , wherein the bitstream is provided by compressing a down-sampled image data generated by down-sampling the input image by a down-sampling module trained based on the same loss function associated with the plurality of analysis models.
16 . The method of claim 14 , wherein the loss function comprises a contour loss function, a plurality of feature map distortions respectively associated with the analysis models, a plurality of analysis loss functions respectively associated with the analysis models, or any combinations thereof.