Granted Patent B2

US 12,382,068 · App. 18/949,892 · Granted Aug 5, 2025

Inventors: Emilien Dupont (London, GB); Hyun Jik Kim (London, GB); Matthias Stephan Bauer (London, GB); Lucas Marvin Theis (London, GB)

Assignee: DeepMind Technologies Limited

H04N19/189H04N19/119H04N19/124H04N19/167H04N19/30

Abstract

Claims (85)

1. A method of encoding input data performed by one or more data processing apparatus, the input data comprising input data values corresponding to respective input data grid points of an input data grid, the method comprising:

optimizing an objective function by jointly optimizing parameters of a synthesis neural network, parameters of a decoder neural network, and a set of respective latent values, each respective latent value corresponding to a respective latent grid point of a respective one of a plurality of latent grids having different respective resolutions, and wherein the optimizing comprises, at each of a plurality of optimization iterations:

generating a respective updated value for each of the respective latent values, comprising:

generating a respective noise distribution for the optimization iteration, wherein the respective noise distribution for the optimization iteration has a shape that is more uniform than the respective noise distribution for a preceding optimization iteration;

sampling a respective noise value for each respective latent value from the respective noise distribution for the optimization iteration;

updating each of the respective latent values by adding the respective noise value for the respective latent value to the respective latent value;

processing the respective updated values using the synthesis neural network to generate respective reconstructed data values, each corresponding to a respective input data value of the input data values;

determining a respective probability distribution for each respective updated value using the decoder neural network;

evaluating the objective function by determining (i) an error between the input data values and the respective reconstructed data values and (ii) a compressibility term based on the respective probability distributions;

determining gradients of the objective function with respect to the parameters of the synthesis neural network, the parameters of the decoder neural network, and the set of respective latent values; and

using the gradients to update one or more of: the parameters of the synthesis neural network, the parameters of the decoder neural network, and the respective latent values;

quantizing the optimized respective latent values; and

encoding the quantized respective latent values using a probability distribution for the respective latent values, the probability distribution being defined by the decoder neural network.

2. The method according to claim 1 , wherein generating a respective updated value for each of the respective latent values comprises:

prior to updating each of the respective latent values by adding the respective noise value for the respective latent value to the respective latent value, applying a soft-rounding function to each of the respective latent values to update each respective latent value.

3. The method according to claim 1 , wherein the respective noise distribution for the optimization iteration is non-uniform.

4. The method according to claim 3 , wherein generating the respective noise distribution for the optimization iteration comprises determining the shape of the respective noise distribution based on a shape parameter that controls the shape of the respective noise distribution.

5. The method according to claim 2 , wherein the soft-rounding function depends on a temperature parameter that controls a smoothness of the soft-rounding function, the temperature parameter being adjusted between the optimization iterations.

6. The method according to claim 1 , wherein:

using the gradients comprises multiplying the gradients by a learning rate before updating the one or more of: the parameters of the synthesis neural network, the parameters of the decoder neural network, and the respective latent values; and

the learning rate varies between the optimization iterations according to a cosine schedule.

7. The method according to claim 1 , wherein the optimizing further comprises, at each of a plurality of further optimization iterations:

quantizing the respective latent values using a further hard-rounding function;

using a soft-rounding estimator to determine further gradients of the objective function using the quantized respective latent values, wherein the soft-rounding estimator provides a smooth approximation to the gradient of the further hard-rounding function; and

using the further gradients determined using the soft-rounding estimator to update one or more of: the parameters of the synthesis neural network, the parameters of the decoder neural network, and the respective latent values.

8. The method according to claim 7 , wherein the soft-rounding estimator depends on a temperature parameter that controls the smoothness of the gradient of a soft-rounding function corresponding to the soft-rounding estimator, the temperature parameter being adjusted between the further optimization iterations such that the gradient of the soft-rounding function increasingly resembles the gradient of the further hard-rounding function.

9. The method according to claim 7 , wherein:

using the further gradients comprises multiplying the further gradients by a further learning rate before updating the one or more of: the parameters of the synthesis neural network, the parameters of the decoder neural network, and the respective latent values; and

the further learning rate is decreased between the further optimization iterations.

10. The method according to claim 7 , wherein the further hard-rounding function quantizes the respective latent values in steps smaller than the steps used by a hard-rounding function for quantizing the respective latent values after the optimizing.

11. The method according to claim 9 , wherein the further hard-rounding function quantizes the respective latent values in steps smaller than one.

12. The method according to claim 1 , wherein the compressibility term is based on a respective probability for each respective latent value, and wherein the respective probabilities are determined by, for each of the respective latent grid points of each of the latent grids:

determining a causal subset of the respective latent values comprising respective latent values corresponding to respective latent grid points that precede the respective latent grid point in the latent grid; and

using the decoder neural network to determine, from the respective latent values in the causal subset, one or more probability distribution parameters defining a conditional probability distribution for a respective latent value at the respective latent grid point; and

using the conditional probability distribution defined by the one or more probability distribution parameters to determine the respective probability of the respective latent value conditioned on the respective latent values in the causal subset.

13. The method according to claim 12 , wherein for one or more of the latent grids, the causal subsets of the respective latent values for each of the respective latent grid points of the latent grid comprise respective latent values of another of the latent grids that has a resolution that is less than a resolution of the latent grid.

14. The method according to claim 13 , wherein the other latent grid immediately precedes the latent grid when the latent grids are arranged in ascending order of resolution.

15. The method according to claim 12 , wherein the decoder neural network comprises a plurality of decoder subnetworks, each decoder subnetwork being configured to process the respective latent values of a respective one of the latent grids.

16. The method according to claim 15 , wherein each decoder subnetwork processes the respective latent values of the respective one of the latent grids independently of the respective latent values of the other latent grids.

17. The method according to claim 12 , wherein the decoder neural network comprises an input layer, an output layer, and intermediate layers between the input and output layers, the intermediate layers comprising activation functions for determining features of the respective latent values at different resolutions, wherein one or more of the intermediate layers comprises a modulation layer configured to apply a transformation to features of respective latent values at a first resolution conditioned on features of respective latent values at a second resolution.

18. The method according to claim 12 , wherein output values of the decoder neural network are exponentiated to determine the one or more probability distribution parameters and wherein a pre-determined shift value is added to the output values prior to exponentiation.

19. The method according to claim 1 , wherein one or more of the synthesis neural network or the decoder neural network comprises activation functions that have a higher computational complexity than Rectified Linear Units.

20. The method according to claim 1 , wherein determining (i) an error between the input data values and the respective reconstructed data values and (ii) a compressibility term based on the respective probability distributions comprises determining the respective reconstructed data values by:

for each of the latent grids, generating from the respective latent values of the latent grid, upsampled latent data comprising respective upsampled latent values for the input data grid points; and

using the synthesis neural network to determine, from the upsampled latent data for each of the latent grids, the respective reconstructed data values for the input data grid points.

21. The method according to claim 1 , wherein each of the input data grid points corresponds to a respective time interval or frequency component of an audio waveform and the input data values correspond to amplitudes of the audio waveform.

22. The method according to claim 1 , wherein each of the input data grid points corresponds to a respective pixel location in one or more images or part of one or more images.

23. The method according to claim 1 , wherein each of the input data grid points corresponds to a respective frame in a sequence of image frames of a video and a respective pixel location in the respective frame.

24. The method according to claim 23 , further comprising:

dividing the video into a plurality of video patches, each video patch corresponding to one or more of: a proper subset of the respective pixel locations of each image frame or a proper subset of the image frames of the video; and wherein the input data comprises one of the plurality of video patches.

25. The method according to claim 23 , wherein the compressibility term is based on a respective probability for each respective latent value, and wherein the respective probabilities are determined by, for each of the respective latent grid points of each of the latent grids:

determining a causal subset of the respective latent values comprising respective latent values corresponding to respective latent grid points that precede the respective latent grid point in the latent grid; and

using the decoder neural network to determine, from the respective latent values in the causal subset, one or more probability distribution parameters defining a conditional probability distribution for a respective latent value at the respective latent grid point; and

using the conditional probability distribution defined by the one or more probability distribution parameters to determine the respective probability of the respective latent value conditioned on the respective latent values in the causal subset, and wherein the causal subset of the respective latent values for each of the respective latent grid points comprises selected latent values of another latent grid corresponding to an image frame of the video that precedes the image frame corresponding to the respective latent grid point in the sequence of image frames.

26. The method according to claim 1 , wherein the optimizing is performed for input data corresponding to a plurality of training examples to generate a corresponding synthesis neural network, decoder neural network and set of respective latent values for each of the one or more training examples, wherein at least some of the parameters of one or more of the synthesis neural networks or the decoder neural networks are shared between the training examples.

27. The method according to claim 26 , wherein the input data of the plurality of training examples correspond to different respective images or videos, or to parts of one image or video.

28. The method according to claim 1 , further comprising providing the encoded latent values in a bitstream.

29. A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform a method of encoding input data, the input data comprising input data values corresponding to respective input data grid points of an input data grid, the method comprising:

optimizing an objective function by jointly optimizing parameters of a synthesis neural network, parameters of a decoder neural network, and a set of respective latent values, each respective latent values corresponding to a respective latent grid point of a respective one of a plurality of latent grids having different respective resolutions, and wherein the optimizing comprises, at each of a plurality of optimization iterations:

generating a respective updated value for each of the respective latent values, comprising:

generating a respective noise distribution for the optimization iteration, wherein the respective noise distribution for the optimization iteration has a shape that is more uniform than the respective noise distribution for a preceding optimization iteration;

sampling a respective noise value for each respective latent value from the respective noise distribution for the optimization iteration;

updating each of the respective latent values by adding the respective noise value for the respective latent value to the respective latent value;

processing the respective updated values using the synthesis neural network to generate respective reconstructed data values, each corresponding to a respective input data value of the input data values;

determining a respective probability distribution for each respective updated value using the decoder neural network;

evaluating the objective function by determining (i) an error between the input data values and the respective reconstructed data values and (ii) a compressibility term based on the respective probability distributions;

determining gradients of the objective function with respect to the parameters of the synthesis neural network, the parameters of the decoder neural network, and the set of respective latent values; and

using the gradients to update one or more of: the parameters of the synthesis neural network, the parameters of the decoder neural network, and the respective latent values;

quantizing the optimized respective latent values; and

encoding the quantized respective latent values using a probability distribution for the latent values, the probability distribution being defined by the decoder neural network.

30. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for encoding input data performed by one or more data processing apparatus, the input data comprising input data values corresponding to respective input data grid points of an input data grid, the operations comprising:

optimizing an objective function by jointly optimizing parameters of a synthesis neural network, parameters of a decoder neural network, and a set of respective latent values, each respective latent value corresponding to a respective latent grid point of a respective one of a plurality of latent grids having different respective resolutions, and wherein the optimizing comprises, at each of a plurality of optimization iterations:

generating a respective updated value for each of the respective latent values, comprising:

generating a respective noise distribution for the optimization iteration, wherein the respective noise distribution for the optimization iteration has a shape that is more uniform than the respective noise distribution for a preceding optimization iteration;

sampling a respective noise value for each respective latent value from the respective noise distribution for the optimization iteration;

updating each of the respective latent values by adding the respective noise value for the respective latent value to the respective latent value;

processing the respective updated values using the synthesis neural network to generate respective reconstructed data values, each corresponding to a respective input data value of the input data values;

determining a respective probability distribution for each respective updated value using the decoder neural network;

evaluating the objective function by determining (i) an error between the input data values and the respective reconstructed data values and (ii) a compressibility term based on the respective probability distributions;

determining gradients of the objective function with respect to the parameters of the synthesis neural network, the parameters of the decoder neural network, and the set of respective latent values; and

using the gradients to update one or more of: the parameters of the synthesis neural network, the parameters of the decoder neural network, and the respective latent values;

quantizing the optimized respective latent values; and

encoding the quantized respective latent values using a probability distribution for the respective latent values, the probability distribution being defined by the decoder neural network.

Assignments (2)

Continuity (2)

References Cited (159)