Image compression and decoding, video compression and decoding: methods and systems
There is disclosed a computer-implemented method for lossy image or video compression, transmission and decoding, the method including the steps of: (i) receiving an input image at a first computer system; (ii) encoding the input image using a first trained neural network, using the first computer system, to produce a latent representation; (iii) quantizing the latent representation using the first computer system to produce a quantized latent; (iv) entropy encoding the quantized latent into a bitstream, using the first computer system; (v) transmitting the bitstream to a second computer system; (vi) the second computer system entropy decoding the bitstream to produce the quantized latent; (vii) the second computer system using a second trained neural network to produce an output image from the quantized latent, wherein the output image is an approximation of the input image. Related computer-implemented methods, systems, computer-implemented training methods and computer program products are disclosed.
1. A computer-implemented method for lossy image or video compression, transmission and decoding, the method including the steps of:
(i) receiving a first input image and a second input image at a first computer system;
(ii) encoding the first input image and the second input image using a first trained neural network, using the first computer system, to produce a latent representation;
(iii) quantizing the latent representation using the first computer system to produce a quantized latent;
(iv) entropy encoding the quantized latent into a bitstream, using the first computer system;
(v) transmitting the bitstream to a second computer system;
(vi) the second computer system entropy decoding the bitstream to produce the quantized latent;
(vii) the second computer system using a second trained neural network to produce an output image from the quantized latent, wherein the output image is an approximation of the input image;
wherein one or more of steps (i)-(vii) comprise the use of an iterative solving method, and
wherein in step (vi), producing the quantized latent comprises, by the second computer system, processing at least part of the entropy decoded bitstream by applying an iterative solving method to said at least part of the entropy decoded bitstream, said at least part of the entropy decoded bitstream comprising spatial and temporal information associated with the first input image and the second input image.
2. The method of claim 1 , the method including use of the iterative solving method to speed up computation relating to one or more probabilistic models used in any of steps (i)-(vii).
3. The method of claim 1 , wherein, in step (vi), producing the quantized latent comprises, by the second computer system, processing at least part of the entropy decoded bitstream by applying the iterative solving method to said at least part of the entropy decoded bitstream.
4. The method of claim 3 , wherein said at least part of the entropy decoded bitstream comprises data indicative of one or more parameters associated with a distribution of the latent representation.
5. The method of claim 4 , wherein said one or more parameters comprise one or more of a mean parameter or a variance parameter of the distribution of the latent representation.
6. The method of claim 3 , wherein said at least part of the entropy decoded bitstream comprises side-information associated with the latent representation.
7. The method of claim 2 , wherein the one or more probabilistic models include one or more autoregressive models.
8. The method of claim 7 , in which the one or more autoregressive models comprise one or more of an intraprediction model, a neural intraprediction model, a block-level model, a filter-bank model, a parameter from a neural network model, a parameter derived from side-information model, a latent variables model, or a temporal modelling model.
9. The method of claim 2 , wherein the one or more probabilistic models include non-autoregressive models.
10. The method of claim 9 , in which the non-autoregressive model is a conditional probabilities from a joint distribution model.
11. The method of claim 10 , wherein the joint distribution model is a standard multivariate distribution model.
12. The method of claim 10 , wherein the joint distribution model is a Markov Random Field model.
13. The method of claim 9 , in which the non-autoregressive model is a generic conditional probability model, or a dependency network.
14. The method of claim 1 , the method including use of an iterative solving method for performing fixed point evaluations in any of steps (i)-(vii).
15. The method of claim 1 , wherein one or more of steps (i)-(vii) use a factorized distribution, in the form of a product of conditional distributions.
16. The method of claim 1 , wherein said iterative solving method is used to solve a system of equations with a triangular structure used in one or more of steps (i)-(vii).
17. A computer implemented method of training a first neural network and a second neural network, the neural networks being for use in lossy image or video compression, transmission and decoding, the method including the steps of:
(i) receiving first and second input training images;
(ii) encoding the first and second input training images using the first neural network, to produce a latent representation;
(iii) quantizing the latent representation to produce a quantized latent;
(iv) using the second neural network to produce an output image from the quantized latent, wherein the output image is an approximation of the input image;
(v) evaluating a loss function based on differences between the output image and the input training image;
(vi) evaluating a gradient of the loss function;
(vii) back-propagating the gradient of the loss function through the second neural network and through the first neural network, to update weights of the second neural network and of the first neural network; and
(viii) repeating steps (i) to (vii) using a set of training images, to produce a trained first neural network and a trained second neural network, and
(ix) storing the weights of the trained first neural network and of the trained second neural network;
wherein one or more of steps (i)-(ix) comprise the use of an iterative solving method, and
wherein in step (vi), producing the quantized latent comprises, by the second computer system, processing at least part of the entropy decoded bitstream by applying an iterative solving method to said at least part of the entropy decoded bitstream, said at least part of the entropy decoded bitstream comprising spatial and temporal information associated with the first input training image and the second input training image.
18. The method of claim 17 , in which the iterative solving method is used for speeding up computation relating to an autoregressive model, or a non-autoregressive model used in any of steps (i)-(ix).
19. The method of claim 17 , comprising performing an automatic differentiation method to backpropagate loss gradients through calculations of said iterative solving method.
20. The method of claim 17 , wherein the gradient of the loss function is approximated and learned using a proxy-function including a neural network.