Granted Patent B2

US 12,432,389 · App. 18/563,734 · Granted Sep 30, 2025

Inventors: George Dan Toderici (Mountain View, CA); Eirikur Thor Agustsson (Zurich, CH); Fabian Julius Mentzer (Zurich, CH); David Charles Minnen (Mountain View, CA); Johannes Balle (San Francisco, CA); Nicholas Johnston (San Jose, CA)

Assignee: Google LLC

H04N19/91G06T3/18G06T5/70H04N19/124H04N19/137G06T2207/20084

Abstract

Claims (76)

1. A method of compressing video performed by a data processing apparatus, comprising:

receiving a video sequence of frames;

generating, using a flow prediction network, an optical flow between two sequential frames, wherein the two sequential frames comprise a first frame and a second frame that is subsequent the first frame;

generating from the optical flow, using a first autoencoder neural network:

a predicted optical flow between the first frame and the second frame; and

a confidence mask that comprises a plurality of confidence values characterizing uncertainty in the predicted optical flow between the first frame and the second frame;

warping a reconstruction of the first frame according to the predicted optical flow and subsequently applying a blurring operation according the confidence mask to obtain an initial predicted reconstruction of the second frame;

generating, using a second autoencoder neural network, a prediction of a residual that is a difference between the second frame and the initial predicted reconstruction of the second frame;

combining the initial predicted reconstruction of the second frame and the prediction of the residual to obtain a predicted second frame;

wherein:

each of the first and second autoencoder neural networks respectively comprise an encoder network and a generator network; and

the generator network of the second autoencoder neural network is a component of a generative adversarial neural network (GANN).

2. The method of claim 1 , wherein:

the first frame and the second frame are subsequent to a third frame, and wherein the third frame is an initial frame in the video sequence; and

further comprising, prior to processing the second and third frames:

generating from the third frame, using a third autoencoder neural network, a predicted reconstruction of the third frame;

generating, using the flow prediction network, an optical flow between third frame and the first frame;

generating from the optical flow, using the first autoencoder neural network:

a predicted optical flow between the third frame and the first frame; and

a confidence mask;

warping the reconstruction of the third frame according to the predicted optical flow and subsequently applying a blurring operation according the confidence mask to obtain an initial predicted reconstruction of the first frame;

generating, using the second autoencoder neural network, a prediction of a residual that is a difference between the first frame and the initial predicted reconstruction of the first frame; and

combining the initial predicted reconstruction of the first frame and the prediction of the residual to obtain a predicted first frame;

wherein:

the third autoencoder neural network comprises an encoder network and a generator network;

the third generator network of the third autoencoder neural network is a component of a generative adversarial neural network (GANN).

3. The method of claim 1 , further comprising:

encoding, using the second autoencoder neural network, a residual to obtain a residual latent;

obtaining, using the third encoder neural network, a free latent by encoding the initial prediction of the second frame; and

concatenating the free latent and the residual latent;

wherein generating, using the second autoencoder neural network, the prediction of the residual comprises generating the predicted residual by the second autoencoder neural network using the concatenation of the free latent and the residual latent.

4. The method of claim 3 , further comprising entropy encoding a quantization of the residual latent, wherein the entropy encoded quantization of the residual latent is included in compressed video data representing the video.

5. The method of claim 3 , wherein encoding the residual to obtain the residual latent comprises:

processing the residual using the encoder neural network of the second autoencoder neural network to generate the residual latent.

6. The method of claim 3 , wherein obtaining the free latent by encoding the initial prediction of the second frame comprises:

processing the initial prediction of the second frame using an encoder neural network to generate the free latent.

7. The method of claim 3 , wherein generating the prediction of the residual comprises:

processing the concatenation of the free latent and the residual latent using the generator neural network of the second autoencoder neural network to generate the prediction of the residual.

8. The method of claim 1 , wherein combining the initial predicted reconstruction of the second frame and the prediction of the residual to obtain the predicted second frame comprises:

generating the predicted second frame by summing the initial predicted reconstruction of the second frame and the prediction of the residual.

9. The method of claim 1 , wherein generating the predicted optical flow between the first frame and the second frame comprises:

processing the optical flow generated by the flow prediction network using the encoder network of the first autoencoder network to generate a flow latent representing the optical flow; and

processing a quantization of the flow latent using the generator neural network of the first autoencoder neural network to generate the predicted optical flow.

10. The method of claim 9 , further comprising entropy encoding the quantization of the flow latent, wherein the entropy encoded quantization of the flow latent is included in compressed video data representing the video.

11. The method of claim 1 , wherein the first and second autoencoder neural networks have been trained on a set of training videos to optimize an objective function that includes an adversarial loss.

12. The method of claim 11 , wherein for one or more video frames of each training video, the adversarial loss is based on a discriminator score, wherein the discriminator score is generated by operations comprising:

generating an input to a discriminator neural network, wherein the input comprises a reconstruction of the video frame that is generated using the first and second autoencoder neural networks; and

providing the input to the discriminator neural network, wherein the discriminator neural network is configured to:

receive an input comprising an input video frame; and

process the input to generate an output discriminator score defining a likelihood that the video frame was generated using the first and second autoencoder neural networks.

13. A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations for compressing video, the operations comprising:

receiving a video sequence of frames;

generating, using a flow prediction network, an optical flow between two sequential frames, wherein the two sequential frames comprise a first frame and a second frame that is subsequent the first frame;

generating from the optical flow, using a first autoencoder neural network:

a predicted optical flow between the first frame and the second frame; and

a confidence mask that comprises a plurality of confidence values characterizing uncertainty in the predicted optical flow between the first frame and the second frame;

warping a reconstruction of the first frame according to the predicted optical flow and subsequently applying a blurring operation according the confidence mask to obtain an initial predicted reconstruction of the second frame;

generating, using a second autoencoder neural network, a prediction of a residual that is a difference between the second frame and the initial predicted reconstruction of the second frame;

combining the initial predicted reconstruction of the second frame and the prediction of the residual to obtain a predicted second frame;

wherein:

each of the first and second autoencoder neural networks respectively comprise an encoder network and a generator network; and

the generator network of the second autoencoder neural network is a component of a generative adversarial neural network (GANN).

14. A system, comprising:

a data processing apparatus; and

a computer storage medium encoded with a computer program, the program comprising instructions that when executed by the data processing apparatus cause the data processing apparatus to perform operations for compressing video, the operations comprising:

receiving a video sequence of frames;

generating, using a flow prediction network, an optical flow between two sequential frames, wherein the two sequential frames comprise a first frame and a second frame that is subsequent the first frame;

generating from the optical flow, using a first autoencoder neural network:

a predicted optical flow between the first frame and the second frame; and

a confidence mask that comprises a plurality of confidence values characterizing uncertainty in the predicted optical flow between the first frame and the second frame;

warping a reconstruction of the first frame according to the predicted optical flow and subsequently applying a blurring operation according the confidence mask to obtain an initial predicted reconstruction of the second frame;

generating, using a second autoencoder neural network, a prediction of a residual that is a difference between the second frame and the initial predicted reconstruction of the second frame;

combining the initial predicted reconstruction of the second frame and the prediction of the residual to obtain a predicted second frame;

wherein:

each of the first and second autoencoder neural networks respectively comprise an encoder network and a generator network; and

the generator network of the second autoencoder neural network is a component of a generative adversarial neural network (GANN).

Assignments (1)

Continuity (2)

References Cited (73)