Image compression and decoding, video compression and decoding: training methods and training systems
A computer-implemented method of training an image generative network f θ for a set of training images, in which an output image {circumflex over (x)} is generated from an input image x of the set of training images non-losslessly, and in which a proxy network is trained for a gradient intractable perceptual metric that evaluates a quality of an output image {circumflex over (x)} given an input image x, the method of training using a plurality of scales for input images from the set of training images. In an embodiment, a blindspot network b α is trained which generates an output image {tilde over (x)} from an input image x. Related computer systems, computer program products and computer-implemented methods of training are disclosed.
1. A computer-implemented method of training an image generative network f θ for a set of training images, in which an output image {circumflex over (x)} is generated from an input image x of the set of training images non-losslessly, and in which a proxy network is trained for a gradient intractable perceptual metric that evaluates a quality of an output image {circumflex over (x)} given an input image x, the method of training using a plurality of scales for input images from the set of training images, the method including the steps of:
(i) receiving an input image x of the set of training images and generating one or more images which are derived from x to make a multiscale set of images {x i } which includes x;
(ii) the image generative network f θ generating an output image {circumflex over (x)} i from an input image x i ϵ{x i }, without tracking gradients for f θ ;
(iii) the proxy network outputting an approximated function output ŷ i , using the x i and the {circumflex over (x)} i as inputs;
(iv) the gradient intractable perceptual metric outputting a function output y i , using the x i and the {circumflex over (x)} i as inputs;
(v) evaluating a loss for the proxy network, using the y i and the ŷ i as inputs, and including the evaluated loss for the proxy network in a loss array for the proxy network;
(vi) repeating steps (ii) to (v) for all the images x i in the multiscale set of images {x i };
(vii) using backpropagation to compute gradients of parameters of the proxy network with respect to an aggregation of the loss array assembled in executions of step (v);
(viii) optimizing the parameters of the proxy network based on the results of step (vii), to provide an optimized proxy network;
(ix) the image generative network f θ generating an output image {circumflex over (x)} i from an input image x i ϵ{x i };
(x) the optimized proxy network outputting an optimized approximated function output ŷ i , using the {circumflex over (x)} i and the {circumflex over (x)} i as inputs;
(xi) evaluating a loss for the generative network f θ , using the x i , the {circumflex over (x)} i and the optimized approximated function output ŷ i as inputs, and including the evaluated loss for the generative network f θ in a loss array for the generative network f θ ;
(xii) repeating steps (ix) to (xi) for all the images x i in the multiscale set of images {x i };
(xiii) using backpropagation to compute gradients of parameters of the generative network f θ with respect to an aggregation of the loss array assembled in executions of step (xi);
(xiv) optimizing the parameters of the generative network f θ based on the results of step (xiii), to provide an optimized generative network f θ , and
(xv) repeating steps (i) to (xiv) for each member of the set of training images.
2. The method of claim 1 , wherein the one or more images which are derived from x to make a multiscale set of images {x i } are derived by downsampling.
3. The method of claim 1 , wherein the generative network f θ includes an encoder, which encodes (by performing lossy encoding) an input image x into a bitstream, and includes a decoder, which decodes the bitstream into an output image {circumflex over (x)}.
4. The method of claim 1 , wherein the method includes an iteration of a training pass of the generative network, and a training pass of the proxy network.
5. The method of claim 1 , wherein the generative and proxy networks have separate optimizers.
6. The method of claim 1 , wherein for the case of proxy network optimization, gradients do not flow through the generative network.
7. The method of claim 1 , wherein the method is used for learned image or video compression.
8. The method of claim 1 , wherein the gradient intractable perceptual metric is a perceptual loss function.
9. The method of claim 1 , wherein the gradient intractable perceptual metric is VMAF, VIF, DLM or IFC, or a mutual information based estimator.
10. The method of claim 1 , wherein the generative network includes a compression network, wherein a term is added to the total loss of the compression network to stabilise the initial training of the compression network.
11. The method of claim 1 , wherein the generative loss includes a generic distortion loss which includes one or more stabilisation terms.
12. The method of claim 1 , wherein the stabilisation terms include Mean Squared Error (MSE) or a combination of analytical losses with weighted deep-embeddings of a pre-trained neural network.
13. The method of claim 1 , wherein a perceptual quality score is assigned to the image at each scale and is aggregated by an aggregation function.
14. The method of claim 1 , wherein the set of images includes a downsampled image that has been downsampled by a factor of two in each dimension.
15. The method of claim 1 , wherein the set of images includes a downsampled image that has been downsampled by a factor of four in each dimension.
16. The method of claim 1 , wherein the mean of the ŷ i is used to train the image generative network by attempting to maximise or minimise the mean of the ŷ i using stochastic gradient descent.
17. The method of claim 1 , wherein the predictions y i are used to train the proxy network to force its predictions to be closer to an output of the perceptual metric, using stochastic gradient descent.
18. The method of claim 1 , wherein for each image x, an RGB image is provided.
19. A computer system configured to train an image generative network f θ for a set of training images, in which the system generates an output image {circumflex over (x)} from an input image x of the set of training images non-losslessly, and in which a proxy network is trained for a gradient intractable perceptual metric that evaluates a quality of an output image {circumflex over (x)} given an input image x, wherein the computer system is configured to:
(i) receive an input image x from the set of training images and generate one or more images which are derived from x to make a multiscale set of images {x i } which includes x;
(ii) use the image generative network f θ to generate an output image {circumflex over (x)} i from an input image x i ϵ{x i }, without tracking gradients for f θ ;
(iii) use the proxy network to output an approximated function output ŷ i , using the x i and the {circumflex over (x)} i as inputs;
(iv) use the gradient intractable perceptual metric to output a function output y i , using the x i and the {circumflex over (x)} i as inputs;
(v) evaluate a loss for the proxy network, using the y i and the ŷ i as inputs, and to include the evaluated loss for the proxy network in a loss array for the proxy network;
(vi) repeat (ii) to (v) for all the images x i in the multiscale set of images {x i };
(vii) use backpropagation to compute gradients of parameters of the proxy network with respect to an aggregation of the loss array assembled in executions of (v);
(viii) optimize the parameters of the proxy network based on the results of (vii), to provide an optimized proxy network;
(ix) use the image generative network f θ to generate an output image {circumflex over (x)} i from an input image x i ϵ{x i };
(x) use the optimized proxy network to output an optimized approximated function output ŷ i , using the x i and the {circumflex over (x)} i as inputs;
(xi) evaluate a loss for the generative network f θ , using the x i , the {circumflex over (x)} i and the optimized approximated function output ŷ i as inputs, and to include the evaluated loss for the generative network f θ in a loss array for the generative network f θ ;
(xii) repeat (ix) to (xi) for all the images x i in the multiscale set of images {x i };
(xiii) use backpropagation to compute gradients of parameters of the generative network f θ with respect to an aggregation of the loss array assembled in executions of (xi);
(xiv) optimize the parameters of the generative network f θ based on the results of (xiii), to provide an optimized generative network f θ , and
(xv) repeat (i) to (xiv) for each member of the set of training images.
20. A computer-implemented method of training an image generative network f θ for a set of training images, in which an output image {circumflex over (x)} is generated from an input image x of the set of training images non-losslessly, and in which a proxy network is trained for a gradient intractable perceptual metric that evaluates a quality of an output image {circumflex over (x)} given an input image x, the method including the steps of:
(i) the image generative network f θ generating an output image {circumflex over (x)} from an input image x of the set of training images, without tracking gradients for f θ ;
(ii) the proxy network outputting an approximated function output ŷ, using x and {circumflex over (x)} as inputs;
(iii) the gradient intractable perceptual metric outputting a function output y, using x and {circumflex over (x)} as inputs;
(iv) evaluating a loss for the proxy network, using y and ŷ as inputs;
(v) using backpropagation to compute gradients of parameters of the proxy network with respect to the loss evaluated in step (iv);
(vi) optimizing the parameters of the proxy network based on the results of step (v), to provide an optimized proxy network;
(vii) the image generative network f θ generating an output image {circumflex over (x)} from an input image x,
(viii) the optimized proxy network outputting an optimized approximated function output ŷ, using x and {circumflex over (x)} as inputs;
(ix) evaluating a loss for the generative network f θ , using x, {circumflex over (x)} and the optimized approximated function output ŷ as inputs;
(x) using backpropagation to compute gradients of parameters of the generative network f θ with respect to the loss evaluated in step (ix);
(xi) optimizing the parameters of the generative network f θ based on the results of step (x), to provide an optimized generative network f θ , and
(xii) repeating steps (i) to (xi) for each member of the set of training images.