Neural image compression with latent feature-domain intra-prediction

1. A method of decoding a neural image with latent feature-domain intra-prediction, the method being performed by at least one processor, and the method comprising:

receiving a set of latent blocks, and for each of the blocks in the set of latent blocks:

predicting a block, based on a set of previously recovered blocks, using a first neural network;

receiving a selection signal indicating a currently recovered block;

based on the received selection signal, performing one of (1) and (2):

(1) generating a compact residual, a set of residual context parameters, and a decoded residual; and

generating a first decoded block, based on the predicted block and the decoded residual;

(2) generating a second decoded block, based on a compact representation block and a set of context parameters;

generating a set of recovered blocks comprising each of the currently recovered blocks;

generating a recovered latent image by merging all the blocks in the set of recovered blocks; and

decoding the generated recovered latent image, using a second neural network, to obtain a reconstructed image.

2. The method of claim 1 , further comprising:

generating a latent representation of an input image using a third neural network; and

partitioning the latent representation into the set of latent blocks.

3. The method of claim 1 , further comprising:

generating the compact residual by quantizing and then entropy encoding a prediction residual;

generating the set of residual context parameters, based on a prediction residual, using a fourth neural network; and

generating the decoded residual by decoding and then dequantizing the compact residual and the residual context parameters,

wherein the compact residual and the set of residual context parameters are generated simultaneously.

4. The method of claim 3 , further comprising:

generating the compact representation block by quantizing and then entropy encoding a current latent block from the set of latent blocks; and

generating the set of context parameters, based on the current latent block, using a fifth neural network,

wherein the set of context parameters and the compact representation are generated simultaneously.

5. The method of claim 4 , wherein the prediction residual is the difference between the predicated block and the current latent block from the set of latent blocks.

6. The method of claim 1 , wherein the set of recovered blocks is used to update the set of previously recovered blocks.

7. The method of claim 1 , further comprising:

the selection signal selecting at least one of the first decoded block and the second decoded block as the currently recovered block;

sending, when the first decoded block is selected, the compact residual, the set of residual context parameters, and the selection signal to a decoder as inputs; and

sending, when the second decoded block is selected, the compact representation, the set of context parameters, and the selection signal to the decoder as inputs.

8. An apparatus for neural image compression with latent feature-domain intra-prediction, the apparatus comprising:

at least one memory configured to store program code; and

at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising:

receiving code configured to cause the at least one processor to receive a set of latent blocks;

prediction code configured to cause the at least one processor to predict a block, based on a set of previously recovered blocks, using a second neural network;

selecting code configured to cause the at least one processor to receive a selection signal indicating a currently recovered block for each of the blocks in the set of latent blocks, based on the received selection signal, perform one of (1) and (2):

(1) first generating code configured to cause the at least one processor to generate a compact residual;

second generating code configured to cause the at least one processor to generate a set of residual context parameters;

third generating code configured to cause the at least one processor to generate a decoded residual; and

first decoding code configured to cause the at least one processor to generate a first decoded block, based on the predicted block and the decoded residual;

(2) second decoding code configured to cause the at least one processor to generate a second decoded block, based on a compact representation block and a set of context parameters;

recovered block generating code configured to cause the at least one processor to generate a set of recovered blocks comprising each of the currently recovered blocks;

merging code configured to cause the at least one processor to merge all the blocks in the set of recovered blocks to generate a recovered latent image; and

third decoding code configured to cause the at least one processor to decode the generated recovered latent image, using a second neural network, to obtain a reconstructed image.

9. The apparatus of claim 8 , further comprising:

latent image generating code configured to cause the at least one processor to generate a latent representation of an input image using a third neural network; and

partitioning code configured to cause the at least one processor to partition the latent representation of the input image into the set of latent blocks.

10. The apparatus of claim 8 , wherein the program code further comprises:

first generating code configured to cause the at least one processor to generate the compact residual by quantizing and then entropy encoding a prediction residual;

second generating code configured to cause the at least one processor to generate the set of residual context parameters, based on the prediction residual, using a fourth neural network; and

third generating code configured to cause the at least one processor to generate the decoded residual by decoding and then dequantizing the compact residual and the residual context parameters,

wherein the compact residual and the set of residual context parameters are generated simultaneously.

11. The apparatus of claim 10 , wherein the program code further comprises:

compact block generating code configured to cause the at least one processor to generate the compact representation block by quantizing and then entropy encoding a current latent block from the set of latent blocks; and

fourth generating code configured to cause the at least one processor to generate the set of context parameters, based on the current latent block, using a fifth neural network,

wherein the set of context parameters and the compact representation are generated simultaneously.

12. The apparatus of claim 11 , wherein the program code further comprises computing code configured to cause the at least one processor to compute the difference between the predicated block and the current latent block from the set of latent blocks as the prediction residual.

13. The apparatus of claim 8 , wherein the program code further comprises updating code configured to cause the at least one processor to update the set of previously recovered blocks using the set of recovered blocks.

14. The apparatus of claim 8 , wherein the selecting code is further configured to cause the at least one processor to:

select at least one of the first decoded block and the second decoded block as the currently recovered block;

send the compact residual, the set of residual context parameters and the selection signal to the third decoding code when the first decoded block is selected; and

send the compact representation, the set of context parameters and the selection signal to the third decoding code when the second decoded block is selected.

15. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor neural image compression with latent feature-domain intra-prediction, cause the at least one processor to:

receive a set of latent blocks;

predict a block, based on a set of previously recovered blocks, using a second neural network;

receive a selection signal indicating a currently recovered block for each of the blocks in the set of latent blocks, based on the received selection signal, perform one of (1) and (2):

(1) generate a compact residual, a set of residual context parameters, and a decoded residual; and

generate a first decoded block based on the predicted block and the decoded residual;

(2) a second decoded block, based on a compact representation block and a set of context parameters;

generate a set of recovered blocks comprising each of the currently recovered blocks;

merge all the blocks in the set of recovered blocks to generate a recovered latent image; and

decode the generated recovered latent image, using a second neural network, to obtain a reconstructed image.

16. The non-transitory computer-readable medium of claim 15 , wherein the instructions, when executed by the at least one processor, further causes the at least one processor to:

generate a latent representation of an input image using a third neural network; and

partition the latent representation of the input image into the set of latent blocks.

17. The non-transitory computer-readable medium of claim 15 , wherein the instructions, when executed by the at least one processor, further causes the at least one processor to:

generate the compact residual by quantizing and then entropy encoding a prediction residual;

generate the set of residual context parameters, based on the prediction residual, using a fourth neural network; and

generate the decoded residual by decoding and then dequantizing the compact residual and the residual context parameters,

wherein the compact residual and the set of residual context parameters are generated simultaneously.

18. The non-transitory computer-readable medium of claim 17 , wherein the instructions, when executed by the at least one processor, further causes the at least one processor to:

generate the compact representation block by quantizing and then entropy encoding a current latent block from the set of latent blocks; and

generate the set of context parameters, based on the current latent block, using a fifth neural network,

wherein the set of context parameters and the compact representation are generated simultaneously.

19. The non-transitory computer-readable medium of claim 17 , wherein the instructions, when executed by the at least one processor, further causes the at least one processor to update the set of previously recovered blocks using the set of recovered blocks.

20. The non-transitory computer-readable medium of claim 17 , wherein the instructions, when executed by the at least one processor, further causes the at least one processor to:

select at least one of the first decoded block and the second decoded block as the currently recovered block;

send the compact residual, the set of residual context parameters and the selection signal to the third decoding code when the first decoded block is selected; and

send the compact representation, the set of context parameters and the selection signal to the third decoding code when the second decoded block is selected.