Image encoding and decoding, video encoding and decoding: methods, systems and training methods

1. A computer implemented method of training a total neural network including a neural network which modifies encoder-decoder performance, the total neural network parameterized by parameters θ and including encoder and decoder neural networks for use in lossy image or video compression, transmission and decoding, the method including the steps of:

(i) receiving an input training image x;

(ii) encoding the input training image using the encoder neural network, to produce a latent representation;

(iii) quantizing the latent representation to produce a quantized latent ŷ;

(iv) using the decoder neural network to produce an output image {circumflex over (x)} from the quantized latent, wherein the output image is an approximation of the input image;

(v) evaluating an augmented Lagrangian loss function including a Lagrange multiplier λ, based on evaluation of a distortion function of differences between the output image and the input training image, and based on differences between a compression rate function of the quantized latent R(ŷ) and a target compression rate r 0 ;

(vi) calculating total neural network parameter gradients using backpropagation of the evaluated augmented Lagrangian loss function;

(vii) updating the total network parameters θ by performing an optimizer step in which a SGD or SGD-like optimizer optimizes the total network parameters θ based on a learning rate for the network parameters optimizer and the total neural network parameter gradients evaluated in step (vi);

(viii) evaluating gradients for the Lagrange multiplier by evaluating an augmented Lagrangian loss function quadratic term weight μ multiplied by a difference between the compression rate function of the quantized latent and the target compression rate;

(ix) updating the Lagrange multiplier by performing an optimizer step in which a SGD or SGD-like optimizer optimizes the Lagrange multiplier based on a learning rate for the Lagrange multiplier optimizer;

(x) repeating steps (i) to (ix) using a set of training images, to produce a trained set of parameters θ, and

(xi) storing the trained set of parameters θ.

2. The method of claim 1 , including the step of clipping the gradients for the Lagrange multiplier evaluated in step (viii), and including updating the Lagrange multiplier by performing the optimizer step in which the SGD or SGD-like optimizer optimizes the Lagrange multiplier based on the learning rate for the Lagrange multiplier optimizer and the clipped gradients for the Lagrange multiplier.

3. The method of claim 1 , wherein the trained set of parameters θ is a converged set of parameters θ.

4. The method of claim 1 , wherein the augmented Lagrangian loss function is D(x, {circumflex over (x)})+λ(R(ŷ)−r 0 )+μ(R(ŷ)−r 0 ) 2 /2, where D is a function measuring distortion of data reconstruction.

5. The method of claim 1 , wherein step (ix) includes modifying the Lagrange multiplier by the learning rate for the Lagrange multiplier optimizer times the loss function quadratic term weight multiplied by a difference between the compression rate function of the quantized latent and the target compression rate.

6. The method of claim 1 , wherein decoupling the learning rate for the Lagrange multiplier from the loss function quadratic term weight through the introduction of the factor of the learning rate for the Lagrange multiplier optimizer means that the loss function quadratic term weight can be kept small, while the multiplier converges in a reasonable amount of time.

7. The method of claim 1 , wherein the SGD-like optimizer optimizing the Lagrange multiplier is an Adam optimizer.

8. The method of claim 1 , in which the quantized latent compression rate is calculated using a training quantisation function in the forward pass steps (iii) and (iv), and in the backward pass steps (v) to (viii) (i.e. in gradient computation) when updating the total neural network parameters θ, but the quantized latent compression rate is calculated using an inference quantisation function when performing updates to the Augmented Lagrangian Method's Lagrange multiplier steps (viii), and (ix).

9. The method of claim 1 , in which the quantized latent is calculated using an inference quantisation function in the forward pass steps (iii) and (iv), and in the backward pass steps (v) to (viii) (i.e. in gradient computation) the quantized latent is calculated using a training quantisation function.

10. The method of claim 1 , wherein the set of training images used is a set of training images used to train the encoder and the decoder, modified so as to be compressed to the target compression rate; the distortion function in step (v) is reduced by a scale factor s (e.g. approximately two, initially); during training, a running average of scaled distortion is updated; at predefined iterations of training, the scale factor s is adjusted in proportion to (e.g. two times) the running average, and the Lagrange multiplier is modified by a factor in inverse proportion to the adjustment to the scale factor.

11. The method of claim 1 , wherein the quantized latent is represented using a probability distribution of the latent space, the probability distribution including a location parameter and a scale parameter σ, wherein σ is reduced as the computer implemented method of training proceeds, e.g. using a decaying scale threshold.

12. The method of claim 11 , wherein σ is reduced as the computer implemented method of training proceeds, until σ reaches a final value.

13. The method of claim 11 , wherein σ is reduced as the computer implemented method of training proceeds, using a progressively decreasing thresholding value t.

14. The method of claim 13 , wherein σ is reduced as the computer implemented method of training proceeds, decaying t linearly with respect to the number of training iterations.

15. The method of claim 13 , wherein σ is reduced as the computer implemented method of training proceeds, decaying t exponentially with respect to the number of training iterations.

16. The method of any of claim 13 , wherein σ is reduced as the computer implemented method of training proceeds, decaying t with respect to the loss metric.

17. The method of claim 11 , wherein all realisations of σ are mapped with functions of a strictly positive codomain, for a defined domain, such as the softplus operation, or the squaring operation with thresholding, or the absolute value operation with thresholding.

18. The method of claim 1 , wherein the quantized latent is represented using a probability distribution of the latent space, the probability distribution including a location parameter and a scale parameter σ, wherein all realisations of σ are thresholded to a fixed value.

19. A computer program product stored on a non-transitory computer readable medium executable on a processor to train a total neural network including a neural network which modifies encoder-decoder performance, the total neural network parameterized by parameters θ and including encoder and decoder neural networks for use in lossy image or video compression, transmission and decoding, the computer program product stored on a non-transitory computer readable medium executable on the processor to:

(i) receive an input training image x;

(ii) encode the input training image using the encoder neural network, to produce a latent representation;

(iii) quantize the latent representation to produce a quantized latent ŷ;

(iv) use the decoder neural network to produce an output image {circumflex over (x)} from the quantized latent, wherein the output image is an approximation of the input image;

(v) evaluate an augmented Lagrangian loss function including a Lagrange multiplier λ, based on evaluation of a distortion function of differences between the output image and the input training image, and based on differences between a compression rate function of the quantized latent R(ŷ) and a target compression rate r 0 ;

(vi) calculate total neural network parameter gradients using backpropagation of the evaluated augmented Lagrangian loss function;

(vii) update the total network parameters θ by performing an optimizer step in which a SGD or SGD-like optimizer optimizes the total network parameters θ based on a learning rate for the network parameters optimizer and the total neural network parameter gradients evaluated in (vi);

(viii) evaluate gradients for the Lagrange multiplier by evaluating an augmented Lagrangian loss function quadratic term weight μ multiplied by a difference between the compression rate function of the quantized latent and the target compression rate;

(ix) update the Lagrange multiplier by performing an optimizer step in which a SGD or SGD-like optimizer optimizes the Lagrange multiplier based on a learning rate for the Lagrange multiplier optimizer;

(x) repeat (i) to (ix) using a set of training images, to produce a trained set of parameters θ, and

(xi) store the trained set of parameters θ.

20. The computer program product of claim 19 , the computer program product stored on a non-transitory computer readable medium executable on the processor to perform the method of claim 1 .