Image encoding and decoding, video encoding and decoding: methods, systems and training methods

1. A computer implemented method of training a total neural network including a neural network which modifies encoder-decoder performance, the total neural network parameterized by parameters θ and including encoder and decoder neural networks for use in lossy image or video compression, transmission and decoding, the method including the steps of:

(i) receiving an input training image x;

(ii) encoding the input training image using the encoder neural network, to produce a latent representation;

(iii) quantizing the latent representation to produce a quantized latent ŷ;

(iv) using the decoder neural network to produce an output image {circumflex over (x)} from the quantized latent, wherein the output image is an approximation of the input image;

(v) evaluating a Lagrangian loss function including a Lagrange multiplier λ, based on evaluation of a distortion function of differences between the output image and the input training image, and based on differences between a compression rate function of the quantized latent R(ŷ) and a target compression rate r 0 ;

(vi) calculating total neural network parameter gradients using backpropagation of the evaluated Lagrangian loss function;

(vii) updating the total network parameters θ by performing a descent step of the Lagrangian loss function with respect to the total network parameters θ based on the total neural network parameter gradients evaluated in step (vi);

(viii) updating the Lagrange multiplier by performing an ascent step of the Lagrangian loss function with respect to the Lagrange multiplier;

(ix) repeating steps (i) to (viii) using a set of training images, to produce a trained set of parameters θ, and optionally a target compression rate r 0 ; and

(x) storing the trained set of parameters θ.

2. The method of claim 1 , wherein the trained set of parameters θ is a converged set of parameters θ.

3. The method of claim 1 , wherein the augmented Lagrangian loss function is D(x, {circumflex over (x)})+λ(R(ŷ)−r 0 )+μ(R(ŷ)−r 0 ) 2 /2, where D is a function measuring distortion of data reconstruction.

4. The method of claim 1 , wherein the quantized latent is represented using a probability distribution of the latent space, the probability distribution including a location parameter and a scale parameter σ, wherein σ is reduced as the computer implemented method of training proceeds, e.g. using a decaying scale threshold.

5. The method of claim 4 , wherein σ is reduced as the computer implemented method of training proceeds, until σ reaches a final value.

6. The method of claim 4 , wherein σ is reduced as the computer implemented method of training proceeds, using a progressively decreasing thresholding value t.

7. The method of claim 6 , wherein σ is reduced as the computer implemented method of training proceeds, decaying t linearly with respect to the number of training iterations.

8. The method of claim 6 , wherein σ is reduced as the computer implemented method of training proceeds, decaying t exponentially with respect to the number of training iterations.

9. The method of claim 6 , wherein σ is reduced as the computer implemented method of training proceeds, decaying t with respect to the loss metric.

10. The method of claim 4 , wherein all realisations of σ are mapped with functions of a strictly positive codomain, for a defined domain, such as the softplus operation, or the squaring operation with thresholding, or the absolute value operation with thresholding.

11. The method of claim 1 , wherein the quantized latent is represented using a probability distribution of the latent space, the probability distribution including a location parameter and a scale parameter σ, wherein all realisations of σ are thresholded to a fixed value.

12. A computer program product stored on a non-transitory computer readable medium executable on a processor to train a total neural network including a neural network which modifies encoder-decoder performance, the total neural network parameterized by parameters θ and including encoder and decoder neural networks for use in lossy image or video compression, transmission and decoding, the computer program product stored on a non-transitory computer readable medium executable on the processor to:

(i) receive an input training image x;

(ii) encode the input training image using the encoder neural network, to produce a latent representation;

(iii) quantize the latent representation to produce a quantized latent ŷ;

(iv) use the decoder neural network to produce an output image {circumflex over (x)} from the quantized latent, wherein the output image is an approximation of the input image;

(v) evaluate a Lagrangian loss function including a Lagrange multiplier λ, based on evaluation of a distortion function of differences between the output image and the input training image, and based on differences between a compression rate function of the quantized latent R(ŷ) and a target compression rate r 0 ;

(vi) calculate total neural network parameter gradients using backpropagation of the evaluated Lagrangian loss function;

(vii) update the total network parameters θ by performing a descent step of the Lagrangian loss function with respect to the total network parameters θ based on the total neural network parameter gradients evaluated in (vi);

(viii) update the Lagrange multiplier by performing an ascent step of the Lagrangian loss function with respect to the Lagrange multiplier;

(ix) repeat (i) to (viii) using a set of training images, to produce a trained set of parameters θ, and optionally a target compression rate r 0 ; and

(x) store the trained set of parameters θ.