Image compression and decoding, video compression and decoding: methods and systems
There is disclosed a computer-implemented method for lossy image or video compression, transmission and decoding, the method including the steps of: (i) receiving an input image at a first computer system; (ii) encoding the input image using a first trained neural network, using the first computer system, to produce a latent representation; (iii) quantizing the latent representation using the first computer system to produce a quantized latent; (iv) entropy encoding the quantized latent into a bitstream, using the first computer system; (v) transmitting the bitstream to a second computer system; (vi) the second computer system entropy decoding the bitstream to produce the quantized latent; (vii) the second computer system using a second trained neural network to produce an output image from the quantized latent, wherein the output image is an approximation of the input image. Related computer-implemented methods, systems, computer-implemented training methods and computer program products are disclosed.
1. A computer implemented method of training a first neural network and a second neural network, the neural networks being for use in lossy image or video compression, transmission and decoding, the method including the steps of:
(i) receiving an input training image;
(ii) segmenting the input training image into training image segments using a segmentation algorithm;
(iii) encoding the training image segments using the first neural network, to produce a latent representation;
(iv) quantizing the latent representation to produce a quantized latent;
(v) using the second neural network to produce an output image from the quantized latent, wherein the output image is an approximation of the input training image;
(vi) evaluating a loss function based on differences between the output image and the input training image;
(vii) evaluating a gradient of the loss function;
(viii) back-propagating the gradient of the loss function through the second neural network and through the first neural network, to update weights of the second neural network and of the first neural network; and
(ix) repeating steps (i) to (viii) using a set of training images, to produce a trained first neural network and a trained second neural network, and
(x) storing the weights of the trained first neural network and of the trained second neural network
wherein a first distortion loss type of the loss function is applied to a first training image segment and a second distortion loss type that is different from the first distortion loss type is applied to a second training image segment.
2. The method of claim 1 , wherein the loss function is a sum of respective rate and respectively weighted respective distortion, over respective training image segments, of a plurality of training image segments.
3. The method of claim 2 , wherein a higher weight is given to training image segments which relate to human faces.
4. The method of claim 2 , wherein a higher weight is given to training image segments which relate to text.
5. The method of claim 1 , wherein the segmentation algorithm is implemented using a neural network.
6. The method of claim 5 , wherein the segmentation algorithm neural network is trained separately to the first neural network and to the second neural network.
7. The method of claim 5 , wherein the segmentation algorithm neural network is trained end-to-end with the first neural network and the second neural network.
8. The method claim 1 , wherein gradients from a compression network do not affect the segmentation algorithm neural network training, and the segmentation network gradients do not affect the compression network gradients.
9. The method of claim 1 , wherein a training pipeline includes a plurality of Encoder-Decoder pairs, wherein each Encoder-Decoder pair produces patches with a particular loss function which determines the types of compression distortion each compression network produces.
10. The method of claim 1 , wherein the loss function is a sum of respective rate and respectively weighted respective distortion, over respective training image segments, of a plurality of training image colour segments.
11. The method of claim 1 , wherein an adversarial generative adversarial network (GAN) loss is applied for high frequency regions, and an mean squared error (MSE) is applied for low frequency areas.
12. The method of claim 1 , wherein a classifier trained to identify optimal distortion losses for image or video segments is used to train the first neural network and the second neural network.
13. The method of claim 1 , wherein the segmentation algorithm is trained in a bi-level fashion.
14. The method of claim 11 , wherein the segmentation algorithm is trained in a bi-level fashion to selectively apply losses for each segment during training of the first neural network and the second neural network.