Granted Patent B2

US 12,294,720 · App. 17/500,355 · Granted May 6, 2025

Inventors: Sheng Lin (San Jose, CA); Ding Ding (Palo Alto, CA); Wei Jiang (Sunnyvale, CA); Wei Wang (Palo Alto, CA); Xiaozhong Xu (State College, PA); Shan Liu (San Jose, CA)

Assignee: TENCENT AMERICA LLC

H04N19/147H04N19/119H04N19/172H04N19/42H04N19/46

Abstract

Claims (67)

1. A method of substitutional end-to-end (E2E) neural image compression (NIC) using a neural network performed by at least one processor, the method comprising:

receiving an input image to an E2E NIC framework;

mapping from the input image x 0 in a high dimensional space to a bit-stream with length R(x 0 );

mapping the bit-stream with the length R(x 0 ) to a compressed representation ;

determining whether there exists a substitution x′ 0 that is mapped to a substitution compressed representation such that a first distance measurement or loss function between and x 0 is less than between a second distance measurement or loss function between and x 0 ;

when the substitution x′ 0 that is mapped to that is closer to x 0 given the second distance measurement or loss function exists, determining a substitute image based on a training model and a step size,

wherein, the substitute image is different from the input image;

encoding the substitute image to generate a compressed representation of the substitute image; and

outputting, as an encoding of the input image for decoding by a decoder, the compressed representation of the substitute image,

wherein the compressed representation of the substitute image replaces a compressed representation of the input image in the E2E NIC framework.

2. The method according to claim 1 , wherein the substitute image is determined by performing an optimization process of the training model, comprising:

adjusting elements of the input image to generate substitute representations; and

selecting the elements with a least distortion loss between the input image and the substitute representations to use as the substitute image.

3. The method according to claim 1 , wherein the substitute image maps to the input image.

4. The method according to claim 1 , wherein the training model is trained based on the determined step size, a number of updates to the input image, and a distortion loss; and

wherein the step size can be increasing, decreasing, or kept the same for one or more iterations of the training model.

5. The method according to claim 1 , wherein a plurality of substitute images are determined based on a plurality of step sizes,

wherein step size values corresponding to the plurality of step sizes are determined based on a plurality of schedulers, and

wherein a substitution image with a highest compression performance is selected for encoding.

6. The method according to claim 5 , further comprising:

splitting the input image into one or more patches,

wherein each of the one or more patches is assigned a scheduler from the plurality of schedulers.

7. An apparatus for substitutional end-to-end (E2E) neural image compression (NIC) using a neural network, the apparatus comprising:

at least one memory configured to store program code; and

at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising:

receiving code configured to cause the at least one processor to receive an input image to an E2E NIC framework;

mapping code configured to cause the at least one processor to map, from the input image x 0 in a high dimensional space to a bit-stream with length R(x 0 ), and map the bit-stream with the length R(x 0 ) to a compressed representation ;

first determining code configured to cause the at least one processor to determine whether there exists a substitution x′ 0 that is mapped to a substitution compressed representation such that a first distance measurement or loss function between and x 0 is less than between a second distance measurement or loss function between and x 0 ;

second determining code configured to cause the at least one processor to determine, when the substitution x′ 0 that is mapped to that is closer to x 0 given the second distance measurement or loss function exists, a substitute image based on a training model and a step size,

wherein, the substitute image is different from the input image;

encoding code configured to cause the at least one processor to encode the substitute image to generate a compressed representation; and

mapping code configured to cause the at least one processor to output, as an encoding of the input image for decoding by a decoder, the compressed representation of the substitute image,

wherein the compressed representation of the substitute image replaces a compressed representation of the input image in the E2E NIC framework.

8. The apparatus according to claim 7 , wherein the substitute image is determined by performing an optimization process of the training model, comprising:

adjusting code configured to cause the at least one processor to adjust elements of the input image to generate substitute representations; and

selecting code configured to cause the at least one processor to select the elements with a least distortion loss between the input image and the substitute representations to use as the substitute image.

9. The apparatus according to claim 7 , wherein the substitute image maps to the input image.

10. The apparatus according to claim 7 , wherein the training model is trained based on the determined step size, a number of updates to the input image, and a distortion loss; and

wherein the step size can be increasing, decreasing, or kept the same for one or more iterations of the training model.

11. The apparatus according to claim 7 , wherein a plurality of substitute images are determined based on a plurality of step sizes,

wherein step size values corresponding to the plurality of step sizes are determined based on a plurality of schedulers, and

wherein a substitution image with a highest compression performance is selected for encoding.

12. The apparatus according to claim 11 , further comprising:

splitting code configured to cause the at least one processor to split the input image into one or more patches;

wherein each of the one or more patches is assigned a scheduler from the plurality of schedulers.

13. A non-transitory computer readable medium storing instructions that, when executed by at least one processor for substitutional end-to-end (E2E) neural image compression (NIC), cause the at least one processor to:

receive an input image to an E2E NIC framework;

map from the input image x 0 in a high dimensional space to a bit-stream with length R(x 0 );

map the bit-stream with the length R(x 0 ) to a compressed representation ;

determine whether there exists a substitution x′ 0 that is mapped to a substitution compressed representation such that a first distance measurement or loss function between and x 0 is less than between a second distance measurement or loss function between and x 0 ;

when the substitution x′ 0 that is mapped to that is closer to x 0 given the second distance measurement or loss function exists, determine a substitute image based on a training model and a step size,

wherein, the substitute image is different from the input image;

encode the substitute image to generate a compressed representation of the substitute image; and

output, as an encoding of the input image for decoding by a decoder, the compressed representation of the substitute image,

wherein the compressed representation of the substitute image replaces a compressed representation of the input image in the E2E NIC framework.

14. The non-transitory computer readable medium of claim 13 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to performing an optimization process of the training model, comprising:

adjust elements of the input image to generate substitute representations; and

select the elements with a least distortion loss between the input image and the substitute representations to use as the substitute image.

15. The non-transitory computer readable medium of claim 13 , wherein the substitute image maps to the input image.

16. The non-transitory computer readable medium of claim 13 , wherein the training model is trained based on the determined step size, a number of updates to the input image, and a distortion loss; and

wherein the step size can be increasing, decreasing, or kept the same for one or more iterations of the training model.

17. The non-transitory computer readable medium of claim 13 , wherein a plurality of substitute images are determined based on a plurality of step sizes,

wherein step size values corresponding to the plurality of step sizes are determined based on a plurality of schedulers, and

wherein a substitution image with a highest compression performance is selected for encoding.

18. The non-transitory computer readable medium of claim 17 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to:

split the input image into one or more patches,

wherein each of the one or more patches is assigned a scheduler from the plurality of schedulers.

Assignments (1)

Continuity (2)

References Cited (12)