Method and data processing system for lossy image or video encoding, transmission and decoding
A method for lossy video encoding, transmission and decoding, the method comprising the steps of: receiving a first frame and a second frame at a first computer system; determining a first flow between the first frame and the second frame; determining a second flow based on the first frame and the second frame; encoding an input based on the first flow and the second flow using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; decoding the latent representation using a second trained neural network to produce an output flow; and using the output flow to obtain an output frame, wherein the output frame is an approximation of the second frame.
1. A method for lossy video encoding, transmission and decoding, the method comprising the steps of:
receiving a first frame and a second frame at a first computer system;
determining a first flow between the first frame and the second frame;
determining a second flow based on the first frame and the second frame;
encoding an input based on the first flow and the second flow using a first trained neural network to produce a latent representation;
transmitting the latent representation to a second computer system;
decoding the latent representation using a second trained neural network to produce an output flow; and
using the output flow to obtain an output frame, wherein the output frame is an approximation of the second frame;
wherein each of the first flow and the second flow define a transformation of a part of an image from a first position in the first frame to a second position in the second frame.
2. The method of claim 1 , further comprising the step of:
transforming the first frame using the first flow to obtain a warped frame;
wherein the second flow is determined between the warped frame and the second frame.
3. The method of claim 2 , wherein the input based on the first flow and the second flow is the second flow.
4. The method of claim 2 , wherein the step of obtaining the output frame comprises applying the output flow to a frame corresponding to the first frame to obtain an intermediate frame.
5. The method of claim 4 , wherein the step of obtaining the output frame comprises applying the first flow to the intermediate frame to obtain the output frame.
6. The method of claim 1 , wherein the second flow is a point-wise flow; and
the input based on the first flow and the second flow is the first flow subtracted from the second flow.
7. The method of claim 6 , wherein the step of obtaining the output frame comprises adding the first flow to the output flow to obtain a reconstructed flow.
8. The method of claim 7 , wherein the step of obtaining the output frame comprises applying the reconstructed flow to a frame corresponding to the first frame to obtain the output frame.
9. The method of claim 2 , wherein the input based on the first flow and the second flow is the first flow and the second flow.
10. The method of claim 9 , wherein the step of obtaining the output frame comprises applying the output flow to a frame corresponding to the first frame to obtain the output frame.
11. The method of claim 1 , further comprising the step of dividing each of the first frame and the second frame into a plurality of corresponding sub-frames; and
a first flow is determined for each of corresponding sub-frames.
12. The method of claim 1 , wherein the first flow is a linear function.
13. The method of claim 12 , wherein the first flow is determined by one of the following methods: a neural network that receives the first frame and the second frame as input and outputs one or more parameters of a matrix defining the first flow, a linear-least squares fit, a Fourier-Mellin transformation and an optimisation that minimizes the photometric error between the first frame and the second frame.
14. The method of claim 1 , wherein the first flow is a non-linear function.
15. The method of claim 14 , wherein the first flow comprises a frame transformation.
16. The method of claim 14 , wherein the non-linear function is a trained neural network.
17. The method of claim 1 , wherein the first flow is a global flow.
18. The method of claim 1 , wherein the second flow is a local flow.
19. A method of training one or more neural networks, the one or more neural networks being for use in lossy video encoding, transmission and decoding, the method comprising the steps of:
receiving a first frame and a second frame at a first computer system; determining a first flow between the first frame and the second frame;
determining a second flow based on the first frame and the second frame;
encoding an input based on the first flow and the second flow using a first neural network to produce a latent representation;
transmitting the latent representation to a second computer system;
decoding the latent representation using a second neural network to produce an output flow;
using the output flow to obtain an output frame, wherein the output frame is an approximation of the second frame;
evaluating a function based on a difference between the output frame and the second frame;
updating the parameters of the first neural network and the second neural network based on the evaluated function; and
repeating the above steps using a first set of input frames to produce a first trained neural network and a second trained neural network;
wherein each of the first flow and the second flow define a transformation of a part of an image from a first position in the first frame to a second position in the second frame.
20. A data processing system configured to perform the method of claim 1 .