Preprocessing image data
Certain aspects of the present disclosure provide techniques for preprocessing, prior to encoding with an external encoder, image data using a preprocessing network comprising a set of inter-connected weights is provided. At the preprocessing network, image data from one or more images is received. The image data is processed using the preprocessing network to generate an output pixel representation for encoding with the external encoder. The weights of the preprocessing network are trained to optimize a combination of at least one quality score indicative of the quality of the output pixel representation and a rate score indicative of the bits required by the external encoder to encode the output pixel representation.
1. A computer-implemented method of preprocessing, prior to encoding using an external encoder, image data using a preprocessing network comprising a set of inter-connected weights, the method comprising:
receiving, at the preprocessing network, image data from one or more images; and
processing the image data using the preprocessing network to generate an output pixel representation for encoding with the external encoder,
wherein the set of inter-connected weights of the preprocessing network are trained to optimize a combination of:
at least one quality score indicative of a quality of the output pixel representation; and
a rate score indicative of a number of bits required by the external encoder to encode the output pixel representation, and
wherein, during an initial setup or training phase, the at least one quality score is optimized in a direction of improved visual quality or reconstruction, and the rate score is optimized in a direction of lower rate.
2. The method according to claim 1 , wherein the at least one quality score is indicative of signal distortion in the output pixel representation.
3. The method according to claim 1 , wherein the at least one quality score is indicative of loss of perceptual or aesthetic quality in the output pixel representation.
4. The method according to claim 1 , wherein a resolution of the output pixel representation is increased or decreased in accordance with an upscaling or downscaling ratio.
5. The method according to claim 1 , further comprising the step of corrupting the output pixel representation by applying one or more mathematically differentiable functions and an approximation, wherein the output pixel representation is corrupted so as to approximate the corruption expected from a block-based transform and quantization used in the external encoder, and/or to approximate the corruption expected from a transform and quantization of errors computed from a block-based temporal prediction process used in the external encoder.
6. The method according to claim 1 , further comprising the step of resizing the output pixel representation to a resolution of the image data using a linear or non-linear filter configured during the initial setup or training phase.
7. The method according to claim 1 , wherein the least one quality score and the rate score are optimized according to a linear or non-linear optimization method that adjusts the set of inter-connected weights of the preprocessing network and/or adjusts a type of architecture used to interconnect the set of inter-connected weights of the preprocessing network.
8. The method according to claim 1 , further comprising the step of encoding the output pixel representation with the external encoder.
9. The method according to claim 1 , wherein the external encoder is an ISO JPEG or ISO MPEG standard encoder, or an AOMedia encoder.
10. The method according to claim 1 , further comprising filtering the output pixel representation using a linear filter, the linear filter comprising a blur or edge-enhancement filter.
11. The method according to claim 1 , wherein the at least one quality score includes one or more of the following: peak-signal-to-noise ratio, structural similarity index metric (SSIM), multiscale quality metrics, detail loss metric or multiscale SSIM, metrics based on multiple quality scores and data-driven learning and training, video multi-method assessment fusion (VMAF), or aesthetic quality metrics.
12. The method according to claim 1 , wherein the at least one quality score and the rate score are combined with linear or non-linear weights, and wherein the linear or non-linear weights are trained based on back-propagation and gradient descent methods with representative training data.
13. A computing device comprising:
a memory comprising computer-executable instructions;
a processor configured to execute the computer-executable instructions and cause the computing device to:
receive, at a preprocessing network, image data from one or more images; and
process the image data using the preprocessing network to generate an output pixel representation for encoding with an external encoder,
wherein a set of inter-connected weights of the preprocessing network are trained to optimize a combination of:
at least one quality score indicative of a quality of the output pixel representation; and
a rate score indicative of a number of bits required by the external encoder to encode the output pixel representation, and
wherein, during an initial setup or training phase, the at least one quality score is optimized in a direction of improved visual quality or reconstruction, and the rate score is optimized in a direction of lower rate.
14. A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by a processor of a computing device, cause the computing device to perform a method, the method comprising:
receiving, at a preprocessing network, image data from one or more images; and
processing the image data using the preprocessing network to generate an output pixel representation for encoding with an external encoder,
wherein a set of inter-connected weights of the preprocessing network are trained to optimize a combination of:
at least one quality score indicative of a quality of the output pixel representation; and
a rate score indicative of a number of bits required by the external encoder to encode the output pixel representation, and
wherein, during an initial setup or training phase, the at least one quality score is optimized in a direction of improved visual quality or reconstruction, and the rate score is optimized in a direction of lower rate.
15. The computing device according to claim 13 , wherein the at least one quality score is indicative of signal distortion in the output pixel representation.
16. The computing device according to claim 13 , wherein the at least one quality score is indicative of loss of perceptual or aesthetic quality in the output pixel representation.
17. The computing device according to claim 13 , wherein a resolution of the output pixel representation is increased or decreased in accordance with an upscaling or downscaling ratio.
18. The non-transitory computer-readable medium according to claim 14 , wherein the at least one quality score is indicative of signal distortion in the output pixel representation.
19. The non-transitory computer-readable medium according to claim 14 , wherein the at least one quality score is indicative of loss of perceptual or aesthetic quality in the output pixel representation.
20. The non-transitory computer-readable medium according to claim 14 , wherein a resolution of the output pixel representation is increased or decreased in accordance with an upscaling or downscaling ratio.