IP Library › Granted Patent US 12,367,387

Granted Patent B2

US 12,367,387 · App. 17/337,820 · Granted Jul 22, 2025

Neural network optimization using curvature estimates based on recent gradients

Inventors: David William Saxton (London, GB); Eshaan Nichani (San Diego, CA)

Assignee: DeepMind Technologies Limited

G06N3/08 G06F17/18 G06N3/047

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12,367,387
App. No.: 17/337,820
Granted: Jul 22, 2025
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a neural network by estimating the objective function curvature based on current and previous gradients. In one aspect, a method comprises: sampling a batch of training data; and for each neural network parameter: determining, based on the current batch of training data, a respective current gradient of the objective function at the current iteration with respect to the current neural network parameter; estimating an objective function curvature with respect to the current neural network parameter based on (i) the current gradient of the objective function at the current iteration, and (ii) a respective previous gradient of the objective function at each of a plurality of previous iterations; and updating a current value of the neural network parameter based on the estimate of the curvature of the objective function.

Claims (70)

1. A method performed by one or more data processing apparatus for training a neural network having a plurality of neural network parameters to perform a machine learning task by optimizing an objective function, the method comprising, at each of a plurality of iterations:

sampling a current batch of training data;

for each neural network parameter of the plurality of neural network parameters:

determining, based on the current batch of training data, a respective current gradient of the objective function at the current iteration with respect to the neural network parameter;

estimating a curvature of the objective function with respect to the neural network parameter based on: (i) the current gradient of the objective function at the current iteration, and (ii) a respective previous gradient of the objective function at each of a plurality of previous iterations, wherein estimating the curvature of the objective function with respect to the neural network parameter comprises:

determining values of parameters of an approximation function that is configured to process a provided value of the neural network parameter to generate an estimate of a gradient of the objective function with respect to the neural network parameter when the neural network parameter has the provided value, and wherein determining the values of the parameters of the approximation function comprises:

determining the values of the parameters of the approximation function to minimize an overall error,

wherein the overall error is based on a respective error for the current iteration and each of the plurality of previous iterations,

wherein the error for each respective iteration measures a difference between: (i) the gradient of the objective function at the iteration, and (ii) an estimate of the gradient of the objective function at the iteration that is generated by processing a value of the neural network parameter at the iteration using the approximation function; and

updating a current value of the neural network parameter based on the estimate of the curvature of the objective function with respect to the neural network parameter at the current iteration;

wherein updating values of the neural network parameters based on estimates of the curvature of the objective function reduces a number of training iterations required for the neural network to achieve a threshold performance on the machine learning task.

2. The method of claim 1 , wherein the curvature of the objective function with respect to the neural network parameter characterizes a second derivative of the objective function with respect to the neural network parameter.

3. The method of claim 1 , wherein the approximation function is a linear function, the parameters of the approximation function comprise a slope parameter and a root parameter, and the slope parameter of the approximation function defines the estimate of the curvature of the objective function with respect to the neural network parameter.

4. The method of claim 3 , wherein processing a provided value of the neural network parameter to generate an estimate of the gradient of the objective function with respect to the neural network parameter when the neural network parameter has the provided value comprises:

subtracting the value of the root parameter from the provided value of the neural network parameter; and

multiplying: (i) the slope parameter, and (ii) a result of the subtraction.

5. The method of claim 1 , wherein the overall error comprises a linear combination of the respective error for the current iteration and each of the plurality of previous iterations, wherein each error is scaled by a respective weight coefficient.

6. The method of claim 5 , further comprising updating a current value of each respective weight coefficient based on a respective value of the gradient of the objective function and a respective value of the neural network parameter at the current iteration and each of the plurality of previous iterations.

7. The method of claim 6 , wherein for each respective weight coefficient, updating the current value of the weight coefficient comprises:

determining a target value of the weight coefficient based on: (i) a variance of the values of the gradient of the objective function, (ii) a variance of the values of the neural network parameter, and (iii) a covariance of the values of the gradient of the objective function and the values of the neural network parameter; and

updating the current value of the weight coefficient based on the target value of the weight coefficient.

8. The method of claim 3 , wherein determining the value of the slope parameter comprises:

determining the value of the slope parameter as a ratio of: (i) a covariance of a respective value of the gradient of the objective function and a respective value of the neural network parameter over the current iteration and each of the plurality of previous iterations, and (ii) a variance of the respective value of the neural network parameter over the current iteration and each of the plurality of previous iterations.

9. The method of claim 8 , wherein determining the value of the root parameter comprises:

determining the value of the root parameter as a difference between:

an expectation of the respective value of the neural network parameter over the current iteration and each of the plurality of previous iterations, and

a ratio of: (i) an expectation of the respective value of the gradient of the objective function over the current iteration and each of the plurality of previous iterations, and (ii) the value of the slope parameter.

10. The method of claim 1 , wherein the error for each iteration is a squared-error between: (i) the gradient of the objective function at the iteration, and (ii) the estimate of the gradient of the objective function at the iteration that is generated by processing the value of the neural network parameter at the iteration using the approximation function.

11. The method of claim 3 , wherein updating the current value of the neural network parameter comprises:

determining a target value of the neural network parameter based on: (i) a sign of the curvature, and (ii) the value of the root parameter; and

updating the current value of the neural network parameter based on the target value of the neural network parameter.

12. The method of claim 11 , wherein the sign of the curvature is non-negative, and determining the target value of the neural network parameter comprises:

determining the target value of the neural network parameter to be equal to the value of the root parameter.

13. The method of claim 12 , further comprising:

determining a trust region, wherein the trust region defines a range of numerical values, comprising:

determining a center of the trust region based on an expectation of the respective value of the neural network parameter over the current iteration and each of the plurality of previous iterations, and

determining a width of the trust region based on a variance of the respective value of the neural network parameter over the current iteration and each of the plurality of previous iterations; and

clipping the target value of the neural network parameter to be included in the trust region.

14. The method of claim 11 , wherein the sign of the curvature is negative, and determining the target value of the neural network parameter comprises:

determining the target value of the neural network parameter such that a distance between: (i) the target value of the neural network parameter, and (ii) the value of the root parameter, is greater than a distance between: (i) the current value of the neural network parameter, and (ii) the value of the root parameter.

15. The method of claim 11 , wherein updating the current value of the neural network parameter based on the target value of the neural network parameter comprises:

setting the updated value of the neural network parameter equal to the target value of the neural network parameter.

16. The method of claim 11 , wherein updating the current value of the neural network parameter based on the target value of the neural network parameter comprises:

sampling the updated value of the neural network parameter from a uniform probability distribution over a range of values that is centered on the target value of the neural network parameter.

17. The method of claim 1 , wherein the neural network is trained to perform an image processing task that comprises processing an image to generate one or more of: a classification of the image, or a regression output characterizing the image, or a segmentation of the image.

18. The method of claim 1 , wherein the neural network is trained to perform an audio processing task that comprises processing audio data to generate one or more of: a classification of the audio data, or a regression output characterizing the audio data, or a segmentation of the audio data.

19. A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a neural network having a plurality of neural network parameters to perform a machine learning task by optimizing an objective function, the operations comprising, at each of a plurality of iterations:

sampling a current batch of training data;

for each neural network parameter of the plurality of neural network parameters:

determining, based on the current batch of training data, a respective current gradient of the objective function at the current iteration with respect to the neural network parameter;

determining the values of the parameters of the approximation function to minimize an overall error,

wherein the overall error is based on a respective error for the current iteration and each of the plurality of previous iterations,

updating a current value of the neural network parameter based on the estimate of the curvature of the objective function with respect to the neural network parameter at the current iteration;

20. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a neural network having a plurality of neural network parameters to perform a machine learning task by optimizing an objective function, the operations comprising, at each of a plurality of iterations:

sampling a current batch of training data;

for each neural network parameter of the plurality of neural network parameters:

determining, based on the current batch of training data, a respective current gradient of the objective function at the current iteration with respect to the neural network parameter;

determining the values of the parameters of the approximation function to minimize an overall error,

wherein the overall error is based on a respective error for the current iteration and each of the plurality of previous iterations,

updating a current value of the neural network parameter based on the estimate of the curvature of the objective function with respect to the neural network parameter at the current iteration;

Assignments (2)

ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 6, 2025

From: DEEPMIND TECHNOLOGIES LIMITED

To: GDM HOLDING LLC

Reel/Frame 071498/0210 →

ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 7, 2021

From: SAXTON, DAVID WILLIAM; NICHANI, ESHAAN

To: DEEPMIND TECHNOLOGIES LIMITED

Reel/Frame 056783/0254 →

Continuity (2)

Provisional Application 63035464 · Jun 5, 2020

Related Publication 20210383222A1 · Dec 9, 2021

References Cited (34)

US 10769528B1 · Wang · 2020 [cited by examiner]

US 20150073755A1 · Tang · 2015 [cited by examiner]

US 20150317126A1 · Lee · 2015 [cited by examiner]

US 20200137394A1 · Shih · 2020 [cited by examiner]

US 20210224634A1 · Chen · 2021 [cited by examiner]

US 20220245519A1 · Miyagawa · 2022 [cited by examiner]

Jamie Wieland, et al. “Stochastic Gradient Estimation using a Single Design Point” “https://www.researchgate.net/publication/221527963_Stochastic_Gradient_Estimation_using_a_Single_Design_Point” (Year: 2006). [cited by examiner]

An, “The effects of adding noise during backpropagation training on a generalization performance,” Neural Computation, Apr. 1996, 8(3):643-674. [cited by applicant]

Baydin et al., “Online learning rate adaptation with hypergradient descent,” CoRR, Feb. 2018, arxiv.org/abs/1703.04782, 11 pages. [cited by applicant]

Berahas et al., “Quasi-newton methods for deep learning: Forget the past, just sample,” CoRR, Sep. 2020, arXiv:1901.09997, 49 pages. [cited by applicant]

Berndt et al., “Estimation and inference in nonlinear structural models,” Annals of Economic and Social Measurement, 1974, 175(3/4):653-665. [cited by applicant]

Blundell et al., “Weight uncertainty in neural networks,” Proceedings of the 32nd International Conference on Machine Learning, 2015, 37:1613-1622. [cited by applicant]

Bottou, “Stochastic gradient learning in neural networks,” Proceedings of Neuro-Nimes, 1991, 91(8):12. [cited by applicant]

Devlin et al., “BERT: Pretraining of deep bidirectional transformers for language understanding,” CoRR, May 2019, arXiv:1810.04805, 16 pages. [cited by applicant]

Dong et al., “On the limited memory BFGS method for large scale optimization,” Mathematical Programming, Aug. 1989, 45(1-3):503-528. [cited by applicant]

Duchi et al., “Adaptive subgradient methods for online learning and stochastic optimization,” Journal of Machine Learning Research, Jul. 2011, 12(7):2121-2159. [cited by applicant]

Graves, “Practical variational inference for neural networks,” Advances in neural information processing systems, 2011, pp. 2348-2356. [cited by applicant]

Guozhong, “The effects of adding noise during backpropagation training on a generalization performance,” Neural Computation, Apr. 1996, 8(3):643-674. [cited by applicant]

He et al., “Deep residual learning for image recognition,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770-778. [cited by applicant]

Ioffe et al., “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” Proceedings of the 32nd International Conference on Machine Learning, 2015, 37:448-456. [cited by applicant]

Jaderberg et al., “Population based training of neural networks,” CoRR, Nov. 2017, arXiv:1711.09846, 21 pages. [cited by applicant]

Kingma et al., “Adam: A method for stochastic optimization,” CoRR, Jan. 2017, arxiv.org/abs/1412.6980, 15 pages. [cited by applicant]

Kirkpatrick et al., “Optimization by simulated annealing,” Science, May 1983, 220(4598):671-680. [cited by applicant]

Liu et al., “On the limited memory BFGS method for large scale optimization,” Mathematical Programming, Aug. 1989, 45(1-3):503-528. [cited by applicant]

Luo et al., “Adaptive gradient methods with dynamic bound of learning rate,” CoRR, Feb. 2019, arxiv.org/abs/1902.09843, 19 pages. [cited by applicant]

Martens et al., “Optimizing neural networks with kronecker-factored approxi mate curvature,” Proceedings of the 32nd International Conference on Machine Learning, 2015, 37:2408-2417. [cited by applicant]

Martens, “Deep learning via hessian-free optimization,” International Concerence on Machine Learning, Jun. 2010, 27:735-742. [cited by applicant]

Neelakantan et al., “Adding gradient noise improves learning for very deep networks,” CoRR, Nov. 2015, arxiv.org/abs/1511.06807, 11 pages. [cited by applicant]

Orabona et al., “Training deep networks without learning rates through coin betting,” CoRR, Nov. 2017, preprint arXiv:1705.07795, 14 pages. [cited by applicant]

Radford et al., “Language models are unsupervised multitask learners,” OpenAI blog, Feb. 2019, 1(8):9. [cited by applicant]

Schaul et al., “No more pesky learning rates,” Proceedings of the 30th International Conference on Machine Learning, 2013, 28(3):343-351. [cited by applicant]

Schneider et al., “DeepOBS: A deep learning optimizer benchmark suite,” CoRR, Mar. 2019, arXiv:1903.05499, 14 pages. [cited by applicant]

Srivastava et al., “Dropout: a simple way to prevent neural networks from overfitting,” The journal of machine learning research, Jan. 2014, 15(1):1929-1958. [cited by applicant]

Such et al., “Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning,” CoRR, Apr. 2018, arxiv.org/abs/1712.06567, 16 pages. [cited by applicant]

Cited By (1)

US 12,585,945