Image encoding and decoding, video encoding and decoding: methods, systems and training methods
Lossy or lossless compression and transmission, comprising the steps of: (i) receiving an input image; (ii) encoding it to produce a y latent representation; (iii) encoding the y latent representation to produce a z hyperlatent representation; (iv) quantizing the z hyperlatent representation to produce a quantized z hyperlatent representation; (v) entropy encoding the quantized z hyperlatent representation into a first bitstream, (vi) processing the quantized z hyperlatent representation to obtain a location entropy parameter μy, an entropy scale parameter σy, and a context matrix Ay of the y latent representation; (vii) processing the y latent representation, the location entropy parameter μy and the context matrix Ay, to obtain quantized latent residuals; (viii) entropy encoding the quantized latent residuals into a second bitstream; and (ix) transmitting the bitstreams.
1 . A computer implemented method of training a neural network for use in lossy image or video compression, the method comprising:
(i) receiving an input image;
(ii) encoding the input image using a first neural network to produce a latent representation, and decoding the latent representation using a second neural network to produce a reconstruction of the input image;
(iii) producing a first feature map, associated with the input image;
(iv) producing a second feature map, associated with the reconstruction of the input image;
(v) evaluating a function based on differences between the first feature map and the second feature map;
(vi) evaluating a gradient of the function;
(vii) back-propagating the gradient of the function through the first neural network and the second neural network to update the weights of the first neural network and the second neural network;
(viii) repeating steps (i) to (vii) to produce a trained first neural network and a trained second neural network.
2 . The method of claim 1 , comprising producing the first feature map and the second feature map using a third neural network.
3 . The method of claim 2 , wherein the third neural network comprises a neural network trained for a task other than image or video compression.
4 . The method of claim 3 , wherein the task other than image or video compression comprises a classification task.
5 . The method of claim 2 , wherein the first feature map and second feature map comprise one or more outputs from one or more layers of the third neural network.
6 . The method of claim 5 , wherein the first feature map and/or the second feature map comprise one or more outputs from one or more intermediate layers of the third neural network.
7 . The method of claim 6 , wherein the third neural network comprises a neural network trained for a task other than for use in lossy image or video compression.
8 . The method of claim 1 , wherein evaluating the function comprises estimating a difference metric between the first feature map and the second feature map.
9 . The method of claim 8 , wherein the difference metric comprises a cosine distance metric.
10 . The method of claim 8 , wherein the difference metric comprises a mean-squared error metric.
11 . The method of claim 8 , wherein the estimating the difference metric comprises estimating differences between the first and second feature maps with the first and second feature maps in a first operand order, and with the first and second feature maps in a second operand order.
12 . The method of claim 1 , wherein the function is further based on differences between the input image and the reconstruction of the input image.
13 . The method of claim 1 , wherein the function defines a first discriminator network and wherein evaluating the function comprises, with the first discriminator network, estimating a probability that the first and/or second feature maps are respectively associated with the input image or the reconstruction of the input image.
14 . The method of claim 13 , comprising:
(ix) back-propagating the gradient of the function through the first discriminator network to update the weights of the discriminator network; and
(x) repeating steps (i) to (vi) and (ix) to produce a trained first discriminator network.
15 . The method of claim 13 , wherein the first and/or second feature maps each comprise first and second tensors of different dimensions, wherein the first discriminator network comprises a plurality of sub-networks, and wherein the method comprises:
with the plurality of sub-networks combining the first and second tensors of different dimensions before estimating said probability.
16 . The method of claim 13 , wherein the first and/or second feature maps comprise one or more outputs from one or more layers of a third neural network, wherein evaluating the function comprises:
with the first discriminator network, estimating a probability that the outputs from a first layer of the third neural network are respectively associated with the input image or the reconstruction of the input image; and
with the second discriminator network, estimating a probability that the outputs from a second layer of the third neural network are respectively associated with the input image or the reconstruction of the input image.
17 . A data processing system configured to perform the method of claim 1 .
18 . A non-transitory computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of claim 1 .
19 . A decoder method comprising:
receiving a bitstream encoded with a latent representation of an image; and
decoding the latent representation using a decoder neural network to reconstruct the image;
wherein the decoder neural network is trained according to the training method of claim 1 , and wherein the training loss function is based on differences between feature maps associated with training images and feature maps associated with corresponding reconstructions of the training images.
20 . A decoder apparatus, the apparatus comprising:
a processor configured to execute a decoder neural network to decode a latent representation of an image to produce a reconstruction of the image;
the decoder neural network trained according to the training method of claim 1 , wherein the training loss function is based on differences between feature maps associated with training images and feature maps associated with corresponding reconstructions of the training images.