IP Library › Granted Patent US 12,541,680
Granted Patent B2
US 12,541,680 · App. 17/169,083 · Granted Feb 3, 2026

Reduced computation real time recurrent learning

Inventors: Jacob Lee Menick (London, GB); Erich Konrad Elsen (San Francisco, CA); Karen Simonyan (London, GB)
Assignee: GDM Holding LLC
G06N3/08G06N3/044
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,541,680
App. No.
17/169,083
Granted
Feb 3, 2026
Kind
B2
Abstract

A computer-implemented method for training a recurrent neural network using forward propagation rather than back propagation through time. The method is particularly suited to training sparse recurrent neural networks, and may be implemented on specialized hardware.

Claims (47)

1 . A computer-implemented method of training a recurrent neural network using forward propagation, the recurrent neural network having a plurality of network parameters and being configured to use hidden state units from a previous time step in computing hidden state units at a current time step, the method comprising:

receiving training data at each of a sequence of time steps and, for a succession of the time steps:

determining, using the training data received at the time step, a dynamics Jacobian matrix for the time step, wherein the dynamics Jacobian matrix defines a derivative of a current hidden state of the recurrent neural network with respect to a previous hidden state of the recurrent neural network from a previous time step;

applying a sparsity mask to a Jacobian matrix for the previous time step to generate a sparsified Jacobian matrix for the previous time step, wherein the Jacobian matrix for the previous time step defines a derivative of the previous hidden state of the recurrent neural network with respect to the network parameters, wherein:

each element of the Jacobian matrix for the previous time step corresponds to a respective hidden state unit of the previous hidden state and to a respective network parameter of the recurrent neural network,

each element defines a derivative of the corresponding hidden state unit of the previous hidden state with respect to the corresponding network parameter, and

the sparsity mask includes a corresponding element for each element of the Jacobian matrix for the preceding time step that is non-zero when a change in the corresponding network parameter would change a value of the corresponding hidden state unit after a single time step of recurrent processing by the recurrent neural network;

determining a Jacobian matrix for the time step by computing a product between the dynamics Jacobian matrix for the time step and the sparsified Jacobian matrix for the previous time step, wherein the Jacobian matrix for the time step defines a derivative of the current hidden state of the recurrent neural network with respect to the network parameters, wherein elements of the Jacobian matrix for the time step each define a derivative of a corresponding dimension of the current hidden state with respect to a corresponding one of the network parameters;

determining a gradient of an optimization function with respect to the network parameters from the Jacobian matrix for the time step; and

adjusting the network parameters dependent on the gradient of the optimization function.

2 . A method as claimed in claim 1 wherein determining the product of the dynamics Jacobian matrix for the time step and the Jacobian matrix for the previous time step comprises performing a sparse matrix multiply.

3 . A method as claimed in claim 1 wherein the corresponding element for each element of the Jacobian matrix for the preceding time step is non-zero when a change in the corresponding network parameter would change a value of the corresponding hidden state unit after no more than two time steps of recurrent processing by the recurrent neural network.

4 . A method as claimed in claim 1 wherein the corresponding element for each element of the Jacobian matrix for the preceding time step is non-zero when a change in the corresponding network parameter would change a value of the corresponding hidden state unit after no more than N time steps of recurrent processing by the recurrent neural network where N is greater than 2.

5 . A method as claimed in claim 1 , further comprising:

imposing sparsity on the Jacobian matrix for the time step, and wherein imposing sparsity on the Jacobian matrix for the time step comprises retaining only elements of the Jacobian matrix for the time step which have one of the top M values.

6 . A method as claimed in claim 1 , wherein determining the Jacobian matrix for the time step further comprises adding an immediate Jacobian matrix for the time step to the product of the dynamics Jacobian matrix for the time step and the Jacobian matrix for the previous time step, wherein the immediate Jacobian matrix for the time step comprises elements which each define a derivative of a dimension of the current hidden state with respect to a current value of one of the network parameters.

7 . A method as claimed in claim 6 wherein elements of the Jacobian matrix for the time step each define a derivative of a dimension of the current hidden state with respect to one of the network parameters, wherein the corresponding element for each element of the Jacobian matrix for the preceding time step is non-zero when a change in the corresponding network parameter would change a value of the corresponding hidden state unit after a single time step of recurrent processing by the recurrent neural network, and wherein the sparsity mask retains an element of J t-1 only if the corresponding element in the immediate Jacobian matrix is non-zero.

8 . A method as claimed in claim 1 , wherein determining the gradient of the optimization function with respect to the network parameters further comprises determining a product of the Jacobian matrix for the time step and a gradient of the optimization function with respect to the current hidden state of the recurrent neural network.

9 . A method as claimed in claim 1 , wherein the dynamics Jacobian matrix is a sparse matrix.

10 . A method as claimed in claim 9 further comprising forming the Jacobian matrix for each time step to exclude rows or columns with zero value, where a row or column with zero value is defined by the sparse dynamics Jacobian matrix.

11 . A method as claimed in claim 1 and configured for implementation on dense matrix multiply-accumulate hardware or a dense matrix processing systolic array, wherein the recurrent neural network is a sparse recurrent neural network such that a parameter matrix is a sparse matrix with a sparsity of 90% or more, wherein the parameter matrix has elements defining, for each dimension of the current hidden state, the network parameters directly influencing the dimensions, and wherein the method uses a dense version of the Jacobian matrix for the time step comprising a version of the Jacobian matrix for the time step in which columns or rows of all zero values are removed.

12 . A method as claimed in claim 1 and configured for implementation in tensor processing hardware, the method comprising:

receiving the training data in a first processor; and

using the tensor processing hardware to determine the gradient of the optimization function by controlling the tensor processing hardware to determine the matrix product of the dynamics Jacobian matrix for the time step and the Jacobian matrix for the previous time step.

13 . A method as claimed in claim 1 wherein the training data comprises a sequence of image pixel data or a sequence of audio data defining a sound.

14 . One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for of training a recurrent neural network using forward propagation, the recurrent neural network having a plurality of network parameters and being configured to use hidden state units from a previous time step in computing hidden state units at a current time step, the operations comprising:

receiving training data at each of a sequence of time steps and, for a succession of the time steps:

determining, using the training data received at the time step, a dynamics Jacobian matrix for the time step, wherein the dynamics Jacobian matrix defines a derivative of a current hidden state of the recurrent neural network with respect to a previous hidden state of the recurrent neural network from a previous time step;

applying a sparsity mask to a Jacobian matrix for the previous time step to generate a sparsified Jacobian matrix for the previous time step, wherein the Jacobian matrix for the previous time step defines a derivative of the previous hidden state of the recurrent neural network with respect to the network parameters, wherein:

each element of the Jacobian matrix for the previous time step corresponds to a respective hidden state unit of the previous hidden state and to a respective network parameter of the recurrent neural network,

each element defines a derivative of the corresponding hidden state unit of the previous hidden state with respect to the corresponding network parameter, and

the sparsity mask includes a corresponding element for each element of the Jacobian matrix for the preceding time step that is non-zero when a change in the corresponding network parameter would change a value of the corresponding hidden state unit after a single time step of recurrent processing by the recurrent neural network;

determining a Jacobian matrix for the time step by computing a product between the dynamics Jacobian matrix for the time step and the sparsified Jacobian matrix for the previous time step, wherein the Jacobian matrix for the time step defines a derivative of the current hidden state of the recurrent neural network with respect to the network parameters, wherein elements of the Jacobian matrix for the time step each define a derivative of a corresponding dimension of the current hidden state with respect to a corresponding one of the network parameters;

determining a gradient of an optimization function with respect to the network parameters from the Jacobian matrix for the time step; and

adjusting the network parameters dependent on the gradient of the optimization function.

15 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a recurrent neural network using forward propagation, the recurrent neural network having a plurality of network parameters and being configured to use hidden state units from a previous time step in computing hidden state units at a current time step, the operations comprising:

receiving training data at each of a sequence of time steps and, for a succession of the time steps:

determining, using the training data received at the time step, a dynamics Jacobian matrix for the time step, wherein the dynamics Jacobian matrix defines a derivative of a current hidden state of the recurrent neural network with respect to a previous hidden state of the recurrent neural network from a previous time step;

applying a sparsity mask to a Jacobian matrix for the previous time step to generate a sparsified Jacobian matrix for the previous time step, wherein the Jacobian matrix for the previous time step defines a derivative of the previous hidden state of the recurrent neural network with respect to the network parameters, wherein:

each element of the Jacobian matrix for the previous time step corresponds to a respective hidden state unit of the previous hidden state and to a respective network parameter of the recurrent neural network,

each element defines a derivative of the corresponding hidden state unit of the previous hidden state with respect to the corresponding network parameter, and

the sparsity mask includes a corresponding element for each element of the Jacobian matrix for the preceding time step that is non-zero when a change in the corresponding network parameter would change a value of the corresponding hidden state unit after a single time step of recurrent processing by the recurrent neural network;

determining a Jacobian matrix for the time step by computing a product between the dynamics Jacobian matrix for the time step and the sparsified Jacobian matrix for the previous time step, wherein the Jacobian matrix for the time step defines a derivative of the current hidden state of the recurrent neural network with respect to the network parameters, wherein elements of the Jacobian matrix for the time step each define a derivative of a corresponding dimension of the current hidden state with respect to a corresponding one of the network parameters;

determining a gradient of an optimization function with respect to the network parameters from the Jacobian matrix for the time step; and

adjusting the network parameters dependent on the gradient of the optimization function.

16 . A system as claimed in claim 15 , wherein determining the Jacobian matrix for the time step further comprises adding an immediate Jacobian matrix for the time step to the product of the dynamics Jacobian matrix for the time step and the Jacobian matrix for the previous time step, wherein the immediate Jacobian matrix for the time step comprises elements which each define a derivative of a dimension of the current hidden state with respect to a current value of one of the network parameters.

17 . A system as claimed in claim 16 , wherein elements of the Jacobian matrix for the time step each define a derivative of a dimension of the current hidden state with respect to one of the network parameters, wherein the corresponding element for each element of the Jacobian matrix for the preceding time step is non-zero when a change in the corresponding network parameter would change a value of the corresponding hidden state unit after a single time step of recurrent processing by the recurrent neural network, and wherein the sparsity mask retains an element of J t-1 only if the corresponding element in the immediate Jacobian matrix is non-zero.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 6, 2025
From: DEEPMIND TECHNOLOGIES LIMITED
To: GDM HOLDING LLC
Reel/Frame 071498/0210 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 16, 2021
From: MENICK, JACOB LEE; ELSEN, ERICH KONRAD; SIMONYAN, KAREN
To: DEEPMIND TECHNOLOGIES LIMITED
Reel/Frame 055267/0993 →
Continuity (2)
Provisional Application 62971566 · Feb 7, 2020
Related Publication 20210256375A1 · Aug 19, 2021
References Cited (32)
US 20170140265A1 · Le · 2017 [cited by examiner]
US 20180260703A1 · Soljacic · 2018 [cited by examiner]
Wand, Xiaozhe, Janusz W.Bialek, Konstantin Turitsyn, “PMU-Based Estimation of Dynamic State Jacobian Matrix and Dynamic System State Matrix in Ambient Conditions”, Jan. 1, 2018, IEEE, p. 682 and 687 (Year: 2018). [cited by examiner]
Wang, Xiaozhe, Janusz W.Bialek, Konstantin Turitsyn, “PMU-Based Estimation of Dynamic State Jacobian Matrix and Dynamic System State Matrix in Ambient Conditions”, Jan. 1, 2018, IEEE, p. 682 and 687 (Year: 2018). [cited by examiner]
Li, Phen, Hongzhi Su, Chengshan Wang, Zhelin Liu, and Jianzhong Wu, “PMU-Based Estimation of Voltage-to-Power Sensitivity for Distribution Networks Considering the Sparsity of Jacobian Matrix”, May 28, 2018, pp. 31308-3… [cited by examiner]
Narang, Sharan, Erich Elsen, Greg Diamos, Shubho Sengupta, “Exploring Sparsity in Recurrent Neural Networks”, 2017, pp. 3-4 and 8 (Year: 2017). [cited by examiner]
Liu, Baoyuan, Min Wang, Hassan Foroosh, Marshall Tappen, Marianna Pensky, “Sparse Convolutional Neural Networks”, p. 810 (Year: 2015). [cited by examiner]
Griewank, Andreas, Jean Utke and Andrea Walther, “Evaluating Higher Derivative Tensors by Forward Propagation of Univariate Taylor Series”, 2000, American Mathematical Society (Year: 2000). [cited by examiner]
Bellec et al., “Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets,” arXiv preprint arXiv:1901.09049, Jan. 2019, 37 pages. [cited by applicant]
Chen et al., “The best of both worlds: Combining recent advances in neural machine translation.” arXiv preprint arXiv:1804.09849, Apr. 2018, 12 pages. [cited by applicant]
Chen et al., “Training deep nets with sublinear memory cost,” arXiv preprint arXiv:1604.06174, Apr. 2016, 12 pages. [cited by applicant]
Cho et al., “Leaming phrase representations using RNN encoder-decoder for statistical machine translation,” arXiv preprint arXiv:1406.1078, Jun. 2014, 15 pages. [cited by applicant]
Clark et al., “Adversarial video generation on complex datasets,” arXiv preprint arXiv:1907.06571, Jul. 2019, 21 pages. [cited by applicant]
Cooijmans et al., “On the variance of unbiased online recurrent optimization,” arXiv preprint arXiv:1902.02405, Feb. 2019, 37 pages. [cited by applicant]
Espeholt et al., “Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures,” International Conference on Machine Learning, Jul. 2018, 10 pages. [cited by applicant]
Gale et al., “The state of sparsity in deep neural networks,” arXiv preprint arXiv:1902.09574, Feb. 2019, 15 pages. [cited by applicant]
Graves et al., “Hybrid computing using a neural network with dynamic external memory,” Nature, Oct. 2016, 538(7626):471-6. [cited by applicant]
Griewank et al., “Algorithm 799: revolve: an implementation of check pointing for the reverse or adjoint mode of computational differentiation,” ACM Transactions on Mathematical Software (TOMS), Mar. 2000, 26(1):19-45. [cited by applicant]
Gruslys et al., “Memory-efficient backpropagation through time,” arXiv preprint arXiv:1606.03401, Jun. 2016, 14 pages. [cited by applicant]
Hochreiter et al., “Long short-term memory,” Neural computation, Nov. 1997, 9(8):1735-80. [cited by applicant]
Itti et al., “A model of saliency-based visual attention for rapid scene analysis,” IEEE Transactions on pattern analysis and machine intelligence, Nov. 1998, 20(11):1254-9. [cited by applicant]
Jaderberg et al., “Human-levelperformance in 3D multiplayer games with population-based reinforcement learning,” Science, May 2019, 364(6443):859-65. [cited by applicant]
Jaeger et al., “The “echo state” approach to analyzing and training recurrent neural networks-with an erratum note,” German National Research Center for Information Technology GMD Technical Report, Jan. 2001, 47 pages. [cited by applicant]
Kalchbrenner et al., Efficient neural audio synthesis, International Conference on Machine Learning, Jul. 2018, 10 pages. [cited by applicant]
Merity et al., “Pointer sentinel mixture models,” arXiv preprint arXiv:1609.07843, Sep. 2016, 13 pages. [cited by applicant]
Mujika et al., “Approximating real-time recurrent learning with random kronecker factors,” Advances in Neural Information Processing Systems, 2018, 31:6594-603. [cited by applicant]
Murray et al., “Local online learning in recurrent networks with random feedback,” Elife, May 2019, 25 pages. [cited by applicant]
Stackman et al., “The synaptic organization of the brain. fifth edition. edited by Gordon M Shepherd,” The Quarterly Review of Biology, Dec. 2005, 80(4):502-502. [cited by applicant]
Tallec et al., “Unbiased online recurrent optimization,” arXiv preprint arXiv:1702.05043, Feb. 2017, 14 pages. [cited by applicant]
Wayne et al., “Unsupervised predictive memory in a goal-directed agent,” arXiv preprint arXiv:1803.10760, Mar. 2018, 57 pages. [cited by applicant]
Werbos et al., “Generalization of backpropagation with application to a recurrent gas market model,” Neural Networks, Jan. 1988, 1(4):339-56. [cited by applicant]
Williams et al., “A learning algorithm for continually running fully recurrent neural networks,” Neural Computation, Jun. 1989, 1(2):270-80. [cited by applicant]