Methods and systems of training neural networks for lossy image or video encoding, transmission and decoding
A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding. The method comprises encoding an input image using a first neural network to produce a latent representation and decoding the latent representation using a second neural network to produce an output image. At least one of the plurality of layers of the first or second neural network comprises a transformation and a function based on an output of the transformation is use to update the parameters of the first neural network and the second neural network.
1 . A method of training one or more neural networks comprising the steps of:
a) receiving an input image at a first computer system;
b) encoding the input image using a first neural network to produce a latent representation;
c) decoding the latent representation using a second neural network to produce an output image, wherein the output image is reconstruction of the input image;
wherein the first neural network and the second neural network each comprise a plurality of layers;
at least one of the plurality of layers of the first or second neural network comprises a transformation, wherein the output of the transformation is a pre-activation feature map;
d) evaluating a function comprising a term which is a function of a difference between the output image and the input image, a term which is the mean of the element-wise squares of the pre-activation latent representation and a term which is a function of the feature map;
e) updating the parameters of the first neural network and the second neural network based on the evaluated function; and
f) repeating the steps a) to e) a plurality of times until the evaluated function reaches a minima using a first set of input images to produce a first trained neural network and a second trained neural network.
2 . The method of claim 1 , wherein the term which is the mean of the element-wise squares of the pre-activation feature map is a sum of the mean of the element-wise squares of a plurality of pre-activation feature maps, wherein each feature map is in a different layer of the plurality of layers of the first or second neural network.
3 . The method of claim 2 , wherein the contribution of each of the sum of the mean of the element-wise squares of the plurality of pre-activation feature maps to the term is scaled by a predetermined value.
4 . The method of claim 1 , wherein the contribution of the term which is the mean of the element-wise squares of the pre-activation feature map to the update to the parameters of the first neural network and the second neural network is scaled by a predetermined value.
5 . The method of claim 1 , wherein the contribution of the term which is the mean of the element-wise squares of the pre-activation feature map to the update to the parameters of the first neural network and the second neural network is scaled by a value; and
the value is additionally updated in at least one of the plurality of repetitions of steps a) to e).
6 . The method of claim 1 , wherein the transformation of the at least one of the plurality of layers of the first or second neural network is an affine linear transformation.
7 . A non-transitory, computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the method of claim 1 .