Method and apparatus for feature substitution for end-to-end image compression

1. A method of feature substitution for end-to-end image compression, the method being performed by at least one processor, and the method comprising:

encoding an input image, using a first neural network, to generate an encoded representation; and

quantizing the generated encoded representation, using a second neural network, to generate a compressed representation,

wherein the first neural network and the second neural network are trained by:

determining a rate loss, based on a bitrate of the generated compressed representation; and

updating the generated encoded representation, based on the determined rate loss.

2. The method of claim 1 , further comprising decoding the generated compressed representation, using a third neural network, to generate an output image.

3. The method of claim 2 , wherein the first neural network, the second neural network and the third neural network are trained by:

determining a distortion loss corresponding to a reconstruction error between the input image and the generated output image; and

updating the generated encoded representation, based on the determined distortion loss, the determined rate loss and an input hyperparameter.

4. The method of claim 3 , wherein the updating the generated encoded representation comprises optimizing an overall loss, based on the following equation:

L ( x,y′,{circumflex over (x)} ′)= R ({circumflex over ( y )}′)+λ D ( x,{circumflex over (x)} ′)

where L(x,y′,{circumflex over (x)}′) is the overall loss, R(ŷ′) is the rate loss determined based on the bitrate of the generated compressed representation ŷ′, λ is the input hyperparameter and D(x, {circumflex over (x)}′) is the distortion loss corresponding to the reconstruction error between the input image x and the generated output image {circumflex over (x)}.

5. The method of claim 2 , wherein the first neural network, the second neural network and the third neural network are trained by:

determining a minimum between a difference between the determined rate loss and a target rate loss, and a predetermined hyperparameter;

determining a distortion loss corresponding to a reconstruction error between the input image and the generated output image; and

updating the generated encoded representation, based on the determined minimum, the determined rate loss and an input hyperparameter.

6. The method of claim 5 , wherein the updating the generated encoded representation comprises optimizing an overall loss, based on the following equation:

L ( x,y′,{circumflex over (x)} ′)=η min( R ({circumflex over ( y )}′)− R target ,ϵ)+ D ( x,{circumflex over (x)} ′),

where L(x,y′,{circumflex over (x)}′) is the overall loss, η is the input hyperparameter, R(ŷ′) is the rate loss determined based on the bitrate of the generated compressed representation ŷ′, R target is the target rate loss, ϵ is the predetermined hyperparameter and D(x,{circumflex over (x)}′) is the distortion loss corresponding to the reconstruction error between the input image x and the generated output image {circumflex over (x)}′.

7. The method of claim 5 , wherein the reconstruction error comprises a peak signal-to-noise ratio (PSNR) and/or a structural similarity index measure (SSIM).

8. An apparatus for feature substitution for end-to-end image compression, the apparatus comprising:

at least one memory configured to store program code; and

at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising:

encoding code configured to cause the at least one processor to encode an input image, using a first neural network, to generate an encoded representation; and

quantizing code configured to cause the at least one processor to quantize the generated encoded representation, using a second neural network, to generate a compressed representation,

wherein the first neural network and the second neural network are trained by:

determining a rate loss, based on a bitrate of the generated compressed representation; and

updating the generated encoded representation, based on the determined rate loss.

9. The apparatus of claim 8 , further comprising decoding code configured to cause the at least one processor to decode the generated compressed representation, using a third neural network, to generate an output image.

10. The apparatus of claim 9 , wherein the first neural network, the second neural network and the third neural network are trained by:

determining a distortion loss corresponding to a reconstruction error between the input image and the generated output image; and

updating the generated encoded representation, based on the determined distortion loss, the determined rate loss and an input hyperparameter.

11. The apparatus of claim 10 , wherein the generated encoded representation is updated by optimizing an overall loss, based on the following equation:

L ( x,y′,{circumflex over (x)} ′)= R ({circumflex over ( y )})+λ D ( x,{circumflex over (x)} ′),

where L(x,y′,{circumflex over (x)}′) is the overall loss, R(ŷ′) is the rate loss determined based on the bitrate of the generated compressed representation ŷ′, λ is the input hyperparameter and D(x,{circumflex over (x)}′) is the distortion loss corresponding to the reconstruction error between the input image x and the generated output image {circumflex over (x)}′.

12. The apparatus of claim 9 , wherein the first neural network, the second neural network and the third neural network are trained by:

determining a minimum between a difference between the determined rate loss and a target rate loss, and a predetermined hyperparameter;

determining a distortion loss corresponding to a reconstruction error between the input image and the generated output image; and

updating the generated encoded representation, based on the determined minimum, the determined rate loss and an input hyperparameter.

13. The apparatus of claim 12 , wherein the generated encoded representation is updated by optimizing an overall loss, based on the following equation:

L ( x,ŷ′,{circumflex over (x)} ′)=η min({circumflex over ( y )}′)− R target ,ϵ)+ D ( x,{circumflex over (x)} ′),

where L(x,y′,{circumflex over (x)}′) is the overall loss, η is the input hyperparameter, R(ŷ′) is the rate loss determined based on the bitrate of the generated compressed representation ŷ′, R target is the target rate loss, ϵ is the predetermined hyperparameter and D(x,{circumflex over (x)}′) is the distortion loss corresponding to the reconstruction error between the input image x and the generated output image {circumflex over (x)}′.

14. The apparatus of claim 12 , wherein the reconstruction error comprises a peak signal-to-noise ratio (PSNR) and/or a structural similarity index measure (SSIM).

15. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor for feature substitution for end-to-end image compression, cause the at least one processor to:

encode an input image, using a first neural network, to generate an encoded representation; and

quantize the generated encoded representation, using a second neural network, to generate a compressed representation,

wherein the first neural network and the second neural network are trained by:

determining a rate loss, based on a bitrate of the generated compressed representation; and

updating the generated encoded representation, based on the determined rate loss.

16. The non-transitory computer-readable medium of claim 15 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to decode the generated compressed representation, using a third neural network, to generate an output image.

17. The non-transitory computer-readable medium of claim 16 , wherein the first neural network, the second neural network and the third neural network are trained by:

determining a distortion loss corresponding to a reconstruction error between the input image and the generated output image; and

updating the generated encoded representation, based on the determined distortion loss, the determined rate loss and an input hyperparameter.

18. The non-transitory computer-readable medium of claim 17 , wherein the generated encoded representation is updated by optimizing an overall loss, based on the following equation:

L ( x,y′,{circumflex over (x)} ′)= R ({circumflex over ( y )})+λ D ( x,{circumflex over (x)} ′),

where L(x,y′,{circumflex over (x)}′) is the overall loss, R(ŷ′) is the rate loss determined based on the bitrate of the generated compressed representation ŷ′, λ is the input hyperparameter and D(x, {circumflex over (x)}′) is the distortion loss corresponding to the reconstruction error between the input image x and the generated output image {circumflex over (x)}′.

19. The non-transitory computer-readable medium of claim 16 , wherein the first neural network, the second neural network and the third neural network are trained by:

determining a minimum between a difference between the determined rate loss and a target rate loss, and a predetermined hyperparameter;

determining a distortion loss corresponding to a reconstruction error between the input image and the generated output image; and

updating the generated encoded representation, based on the determined minimum, the determined rate loss and an input hyperparameter.

20. The non-transitory computer-readable medium of claim 19 , wherein the generated encoded representation is updated by optimizing an overall loss, based on the following equation:

L ( x,y′,{circumflex over (x)} ′)=η min( R ({circumflex over ( y )}′)− R target ,ϵ)+ D ( x,{circumflex over (x)} ′),

where L(x,y′,{circumflex over (x)}′) is the overall loss, η is the input hyperparameter, R(ŷ′) is the rate loss determined based on the bitrate of the generated compressed representation ŷ′, R target is the target rate loss, ϵ is the predetermined hyperparameter and D(x,{circumflex over (x)}′) is the distortion loss corresponding to the reconstruction error between the input image x and the generated output image {circumflex over (x)}′.