IP Library › Granted Patent US 12,561,573
Granted Patent B2
US 12,561,573 · App. 18/497,931 · Granted Feb 24, 2026

Continuous control with deep reinforcement learning

Inventors: Timothy Paul Lillicrap (London, GB); Jonathan James Hunt (London, GB); Alexander Pritzel (London, GB); Nicolas Manfred Otto Heess (London, GB); Tom Erez (London, GB); Yuval Tassa (London, GB); David Silver (Hitchin, GB); Daniel Pieter Wierstra (London, GB)
Assignee: DeepMind Technologies Limited
G06N3/08G06N3/006G06N3/045G06N3/084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,561,573
App. No.
18/497,931
Granted
Feb 24, 2026
Kind
B2
Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an actor neural network used to select actions to be performed by an agent interacting with an environment. One of the methods includes obtaining a minibatch of experience tuples; and updating current values of the parameters of the actor neural network, comprising: for each experience tuple in the minibatch: processing the training observation and the training action in the experience tuple using a critic neural network to determine a neural network output for the experience tuple, and determining a target neural network output for the experience tuple; updating current values of the parameters of the critic neural network using errors between the target neural network outputs and the neural network outputs; and updating the current values of the parameters of the actor neural network using the critic neural network.

Claims (87)

1 . A method performed by one or more computers, the method comprising:

receiving a current observation characterizing a current state of an environment;

processing the current observation using an actor neural network to map an input observation to a point in a continuous action space that defines an action to be performed by an agent in response to the input observation; and

causing the agent to perform the action defined by the point in the continuous action space, wherein the actor neural network has been trained jointly with a critic neural network that is configured to receive a critic input specifying a training observation and a training action and to process the critic input to generate a neural network output that is an estimate of time-discounted total future rewards that will be received in response to the agent performing the training action in response to the training observation, and wherein jointly training the critic neural network and the actor neural network comprises:

obtaining a minibatch of experience tuples from a replay memory, each experience tuple comprising a training observation characterizing a training state of an environment, a training action from a continuous space of actions performed by an agent in response to receiving the training observation, a training reward received by the agent for performing the training action, and a next training observation characterizing a next training state of the environment; and

updating current values of parameters of the actor neural network using the minibatch of experience tuples, wherein the updating comprises:

for each experience tuple in the minibatch:

processing the training observation and the training action in the experience tuple using the critic neural network to determine a neural network output for the experience tuple in accordance with current values of parameters of the critic neural network;

processing the next training observation using a target actor neural network to determine a predicted next action for the experience tuple in accordance with current values of parameters of the target actor neural network, wherein the target actor neural network has a same neural network architecture as the actor neural network but the current values of the parameters of the target actor neural network are different from the current values of the parameters of the actor neural network;

processing the next training observation and the predicted next action for the experience tuple using a target critic neural network to generate a predicted next neural network output in accordance with current values of parameters of the target critic neural network, wherein the target critic neural network has a same neural network architecture as the critic neural network but the current values of the parameters of the target critic neural network are different from the current values of the parameters of the critic neural network; and

determining a target neural network output for the experience tuple from the training reward and the predicted next neural network output for the experience tuple,

updating the current values of the parameters of the critic neural network using errors between the target neural network outputs and the neural network outputs for the experience tuples in the minibatch; and

updating current values of the parameters of the actor neural network using the critic neural network.

2 . The method of claim 1 , wherein jointly training the critic neural network and the actor neural network further comprises:

updating the current values of the parameters of the target actor neural network using the updated values of the parameters of the actor neural network; and

updating the current values of the parameters of the target critic neural network using the updated values of the parameters of the critic neural network.

3 . The method of claim 2 , wherein the current values of the parameters of the target actor neural network and the target critic neural network are constrained to change slowly during the training of the actor neural network.

4 . The method of claim 1 , wherein the critic neural network, the actor neural network, or both include one or more batch normalization neural network layers.

5 . The method of claim 1 , wherein the current observation comprises one or more low-dimensional feature vectors that characterize the current state of the environment.

6 . The method of claim 5 , wherein values of different dimensions of the low-dimensional feature vectors have varying ranges.

7 . The method of claim 1 , wherein the current observation comprises high-dimensional pixel inputs from one or more images that characterize the current state of the environment.

8 . The method of claim 1 , wherein jointly training the critic neural network and the actor neural network further comprises:

independently from updating the current values of the parameters of the actor neural network using the minibatch of experience tuples:

generating a new experience tuple, comprising:

receiving a new training observation,

processing the new training observation using the actor neural network to select a new training action to be performed by the agent in accordance with the current values of the parameters of the actor neural network,

receiving a new training reward in response to the agent performing the new training action,

receiving a new next training observation, and

generating a new experience tuple that includes the new training observation, the new training action, the new training reward, and the new next training observation; and

adding the new experience tuple to the replay memory.

9 . The method of claim 8 , wherein processing the new training observation using the actor neural network to select a new training action comprises:

processing the new training observation using the actor neural network to generate an initial new training action;

sampling from a noise process to obtain a noise factor; and

adjusting the initial new action by the noise factor to generate the new training action.

10 . The method of claim 1 , wherein updating the current values of the parameters of the actor neural network using the critic neural network comprises:

for each experience tuple in the minibatch:

processing the training observation in the experience tuple using the actor neural network in accordance with the current values of the parameters of the actor neural network to generate a next action for the training observation;

determining, by backpropagating through the critic neural network, a first gradient of the critic neural network with respect to the next action taken at the training observation-next action input pair and in accordance with the current values of the parameters of the critic neural network;

determining, by backpropagating through the actor neural network, a second gradient of the actor neural network with respect to the parameters of the actor neural network taken at the training observation and in accordance with current values of the parameters of the actor neural network; and

determining a parameter update for the actor neural network from the first gradient and the second gradient.

11 . A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

receiving a current observation characterizing a current state of an environment;

processing the current observation using an actor neural network to map an input observation to a point in a continuous action space that defines an action to be performed by an agent in response to the input observation; and

causing the agent to perform the action defined by the point in the continuous action space, wherein the actor neural network has been trained jointly with a critic neural network that is configured to receive a critic input specifying a training observation and a training action and to process the critic input to generate a neural network output that is an estimate of time-discounted total future rewards that will be received in response to the agent performing the training action in response to the training observation, and wherein jointly training the critic neural network and the actor neural network comprises:

obtaining a minibatch of experience tuples from a replay memory, each experience tuple comprising a training observation characterizing a training state of an environment, a training action from a continuous space of actions performed by an agent in response to receiving the training observation, a training reward received by the agent for performing the training action, and a next training observation characterizing a next training state of the environment; and

updating current values of parameters of the actor neural network using the minibatch of experience tuples, wherein the updating comprises:

for each experience tuple in the minibatch:

processing the training observation and the training action in the experience tuple using the critic neural network to determine a neural network output for the experience tuple in accordance with current values of parameters of the critic neural network;

processing the next training observation using a target actor neural network to determine a predicted next action for the experience tuple in accordance with current values of parameters of the target actor neural network, wherein the target actor neural network has a same neural network architecture as the actor neural network but the current values of the parameters of the target actor neural network are different from the current values of the parameters of the actor neural network;

processing the next training observation and the predicted next action for the experience tuple using a target critic neural network to generate a predicted next neural network output in accordance with current values of parameters of the target critic neural network, wherein the target critic neural network has a same neural network architecture as the critic neural network but the current values of the parameters of the target critic neural network are different from the current values of the parameters of the critic neural network; and

determining a target neural network output for the experience tuple from the training reward and the predicted next neural network output for the experience tuple,

updating the current values of the parameters of the critic neural network using errors between the target neural network outputs and the neural network outputs for the experience tuples in the minibatch; and

updating current values of the parameters of the actor neural network using the critic neural network.

12 . The system of claim 11 , wherein jointly training the critic neural network and the actor neural network further comprises:

updating the current values of the parameters of the target actor neural network using the updated values of the parameters of the actor neural network; and

updating the current values of the parameters of the target critic neural network using the updated values of the parameters of the critic neural network.

13 . The system of claim 12 , wherein the current values of the parameters of the target actor neural network and the target critic neural network are constrained to change slowly during the training of the actor neural network.

14 . The system of claim 11 , wherein the critic neural network, the actor neural network, or both include one or more batch normalization neural network layers.

15 . The system of claim 11 , wherein the current observation comprises one or more low-dimensional feature vectors that characterize the current state of the environment.

16 . The system of claim 15 , wherein values of different dimensions of the low-dimensional feature vectors have varying ranges.

17 . The system of claim 11 , wherein the current observation comprises high-dimensional pixel inputs from one or more images that characterize the current state of the environment.

18 . The system of claim 11 , wherein jointly training the critic neural network and the actor neural network further comprises:

independently from updating the current values of the parameters of the actor neural network using the minibatch of experience tuples:

generating a new experience tuple, comprising:

receiving a new training observation,

processing the new training observation using the actor neural network to select a new training action to be performed by the agent in accordance with the current values of the parameters of the actor neural network,

receiving a new training reward in response to the agent performing the new training action,

receiving a new next training observation, and

generating a new experience tuple that includes the new training observation, the new training action, the new training reward, and the new next training observation; and

adding the new experience tuple to the replay memory.

19 . The system of claim 18 , wherein processing the new training observation using the actor neural network to select a new training action comprises:

processing the new training observation using the actor neural network to generate an initial new training action;

sampling from a noise process to obtain a noise factor; and

adjusting the initial new action by the noise factor to generate the new training action.

20 . One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving a current observation characterizing a current state of an environment;

processing the current observation using an actor neural network to map an input observation to a point in a continuous action space that defines an action to be performed by an agent in response to the input observation; and

causing the agent to perform the action defined by the point in the continuous action space, wherein the actor neural network has been trained jointly with a critic neural network that is configured to receive a critic input specifying a training observation and a training action and to process the critic input to generate a neural network output that is an estimate of time-discounted total future rewards that will be received in response to the agent performing the training action in response to the training observation, and wherein jointly training the critic neural network and the actor neural network comprises:

obtaining a minibatch of experience tuples from a replay memory, each experience tuple comprising a training observation characterizing a training state of an environment, a training action from a continuous space of actions performed by an agent in response to receiving the training observation, a training reward received by the agent for performing the training action, and a next training observation characterizing a next training state of the environment; and

updating current values of parameters of the actor neural network using the minibatch of experience tuples, wherein the updating comprises:

for each experience tuple in the minibatch:

processing the training observation and the training action in the experience tuple using the critic neural network to determine a neural network output for the experience tuple in accordance with current values of parameters of the critic neural network;

processing the next training observation using a target actor neural network to determine a predicted next action for the experience tuple in accordance with current values of parameters of the target actor neural network, wherein the target actor neural network has a same neural network architecture as the actor neural network but the current values of the parameters of the target actor neural network are different from the current values of the parameters of the actor neural network;

processing the next training observation and the predicted next action for the experience tuple using a target critic neural network to generate a predicted next neural network output in accordance with current values of parameters of the target critic neural network, wherein the target critic neural network has a same neural network architecture as the critic neural network but the current values of the parameters of the target critic neural network are different from the current values of the parameters of the critic neural network; and

determining a target neural network output for the experience tuple from the training reward and the predicted next neural network output for the experience tuple,

updating the current values of the parameters of the critic neural network using errors between the target neural network outputs and the neural network outputs for the experience tuples in the minibatch; and

updating current values of the parameters of the actor neural network using the critic neural network.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 6, 2025
From: DEEPMIND TECHNOLOGIES LIMITED
To: GDM HOLDING LLC
Reel/Frame 071498/0210 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 1, 2023
From: LILLICRAP, TIMOTHY PAUL; HUNT, JONATHAN JAMES; PRITZEL, ALEXANDER; HEESS, NICOLAS MANFRED OTTO; EREZ, TOM; TASSA, YUVAL; SILVER, DAVID; WIERSTRA, DANIEL PIETER
To: GOOGLE INC.
Reel/Frame 065421/0126 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 1, 2023
From: GOOGLE, INC.
To: DEEPMIND TECHNOLOGIES LIMITED
Reel/Frame 065421/0274 →
Continuity (4)
Continuation 17019927 · Sep 14, 2020
Continuation 15217758 · Jul 22, 2016
Provisional Application 62196854 · Jul 24, 2015
Related Publication 20240177002A1 · May 30, 2024
References Cited (80)
US 5479576A · Watanabe · 1995 [cited by examiner]
US 5608843A · Baird, III · 1997 [cited by examiner]
US 6085178A · Bigus et al. · 2000 [cited by applicant]
US 6917925B2 · Berenji et al. · 2005 [cited by applicant]
US 7805388B2 · Weston et al. · 2010 [cited by applicant]
US 7970718B2 · Guyon et al. · 2011 [cited by applicant]
US 8392346B2 · Ueda et al. · 2013 [cited by applicant]
US 9015093B1 · Commons · 2015 [cited by examiner]
US 11514305B1 · Commons · 2022 [cited by examiner]
US 20030074338A1 · Young · 2003 [cited by examiner]
US 20050245303A1 · Graepel et al. · 2005 [cited by applicant]
US 20060050953A1 · Farmer et al. · 2006 [cited by applicant]
US 20070011119A1 · Thaler · 2007 [cited by examiner]
US 20130178953A1 · Wersborg · 2013 [cited by examiner]
US 20130262353A1 · Vamvoudakis et al. · 2013 [cited by applicant]
US 20130325776A1 · Ponulak · 2013 [cited by examiner]
US 20140025613A1 · Ponulak · 2014 [cited by examiner]
US 20150100530A1 · Mnih · 2015 [cited by examiner]
CN 101466111 · 2009 [cited by applicant]
CN 102207928 · 2011 [cited by applicant]
JP 2013081683 · 2013 [cited by applicant]
RU 2542901 · 2015 [cited by applicant]
Q-Learning in Continuous State and Action Spaces Gaskett et al. (Year: 1999). [cited by examiner]
Real-time reinforcement learning by sequential Actor-Critics and experience replay Pawel Wawrzynski (Year: 2009). [cited by examiner]
On-Line Learning Control by Association and Reinforcement Si et al. (Year: 2001). [cited by examiner]
Adaptive-Critic-Based Optimal Neurocontrol for Synchronous Generators in a Power System Using MLP/RBF Neural Networks Park et al. (Year: 2003). [cited by examiner]
An Online Actor-Critic Learning Approach with Levenberg-Marquardt Algorithm Ni et al. (Year: 2011). [cited by examiner]
Deterministic Policy Gradient Algorithms Silver et al. (Year: 2014). [cited by examiner]
Improved Neural Fitted Q Iteration Applied to a Novel Computer Gaming and Learning Benchmark Gabel et al. (Year: 2011). [cited by examiner]
From Pixels to Torques: Policy Learning with Deep Dynamical Models Wahlstrom et al. (Year: 2015). [cited by examiner]
Continuous control with deep reinforcement learning Lillicrap et al (Year: 2015). [cited by examiner]
Application of Actor-Critic Learning To Adaptive State Space Construction Cheng et al. (Year: 2004). [cited by examiner]
Playing Atari with Deep Reinforcement Learning Mnih et al (Year: 2013). [cited by examiner]
Deisenroth et al., “A Survey on Policy Search for Robotics,” Foundations and Trends in Robotics, 2013, 2(1-2):1-142. [cited by applicant]
Deisenroth et al., “Pilco: A model-based and data-efficient approach to policy search,” Proceedings of the 28th International Conference on machine learning (ICML-11), 2011, pp. 465-472. [cited by applicant]
Fremaux et al., Reinforcement Learning Using a Continuous Time Actor-Critic Framework with Spiking Neurons, PLoS Comput. Biol., Apr. 2013, 9(4):e1003024. [cited by applicant]
Glascher et al., “States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning,” Neuron, May 2010, 66(4):585-595. [cited by applicant]
Glorot et al., “Deep sparse rectifier networks,” Proceedings of the 14th International Conference on Artificial Intelligence and Statistics. JMLR W&CP, 2011, 15:315-323. [cited by applicant]
Hafner et al., “Reinforcement learning in feedback control,” Machine Learning, Jul. 2011, 84(1-2):137-169. [cited by applicant]
Hasselt, “Double Q-learning,” Advances in Neural Information Processing Systems, 2010, pp. 2613-2621. [cited by applicant]
Heess et al., “Actor-Critic Reinforcement Learning with Energy-Based Policies,” 10th European Workshop on Reinforcement Learning, JMLR: Workshop and Conference Proceedings, 2012, 24:43-57. [cited by applicant]
International Search Report and Written Opinion in International Application No. PCT/US2016/043716, Oct. 14, 2016, 12 pages. [cited by applicant]
Ioffe et al., “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” CoRR, Mar. 2015, arXiv preprint arXiv:1502.03167, 11 pages. [cited by applicant]
Jia, “Deep Learning for Actor-Critic Reinforcement Learning,” Thesis for the degree of Master of Science in Systems and Control, Delft University of Technology, May 2015, 73 pages. [cited by applicant]
Kingma et al., “Adam: A Method for Stochastic Optimization,” CoRR, Jul. 2015, arXiv preprint arXiv:1412.6980, 15 pages. [cited by applicant]
Kiumarsi et al., “Actor-Critic-Based Optimal Tracking for Partially Unknown Nonlinear Discrete-Time Systems,” IEEE Transactions on Neural Networks and Learning Systems, Jan. 2015, 26(1):140-151. [cited by applicant]
Koutnik et al., “Evolving deep unsupervised convolutional networks for vision-based reinforcement learning,” Proceedings of the 2014 conference on Genetic and evolutionary computation, ACM, Jul. 2014, pp. 541-548. [cited by applicant]
Koutnik et al., “Online Evolution of Deep Convolutional Network for Vision-Based Reinforcement Learning,” From Animals to Animats 13, Springer, 2014, pp. 260-269. [cited by applicant]
Krizhevsky et al., “ImageNet Classification with Deep Convolutional Neural Networks,” Advances in Neural Information Processing Systems, 2012, pp. 1097-1105. [cited by applicant]
Levine et al., “End-to-End Training of Deep Visuomotor Policies,” Journal of Machine Learning Research 17, Apr. 2016, pp. 1-40. [cited by applicant]
Mnih et al., “Playing Atari with Deep Reinforcement Learning,” CoRR, Dec. 19, 2013, arXiv:1312.5602, 9 pages. [cited by applicant]
Mnih et al., “Human-level control through deep reinforcement learning,” Nature, Feb. 2015, 5189(7540):529-533. [cited by applicant]
Office Action in Australian Appln. No. 2016297852, mailed on Nov. 5, 2018, 3 pages. [cited by applicant]
Office Action in Canadian Appln. No. 2,993,551, dated Aug. 19, 2020, 7 pages. [cited by applicant]
Office Action in Chinese Appln. No. 201680043561.7, dated Jun. 29, 2021, 27 pages (with English translation). [cited by applicant]
Office Action in European Appln. No. 16745383.6, dated Feb. 15, 2022, 13 pages. [cited by applicant]
Office Action in European Appln. No. 16745383.6, dated Sep. 7, 2020, 7 pages. [cited by applicant]
Office Action in Great Britain Appln. No. GB1802748.2, dated Jun. 18, 2021, 7 pages. [cited by applicant]
Office Action in Indian Appln. No. 201847005934, dated Mar. 12, 2021, 6 pages (with English translation). [cited by applicant]
Office Action in Israeli Appln. No. 257103, dated Dec. 30, 2019, 6 pages (with English translation). [cited by applicant]
Office Action in Israeli Appln. No. 257103, dated Jan. 17, 2021, 3 pages (with English summary). [cited by applicant]
Office Action in Japanese Appln. No. 2018-523386, dated Jun. 20, 2019, 8 pages (with English translation). [cited by applicant]
Office Action in Korean AppIn. No. 10-2018-7005435, dated Jan. 17, 2020, 7 pages (with English translation). [cited by applicant]
Office Action issued in Canadian Appln. No. 2993551, dated May 21, 2021, 5 pages. [cited by applicant]
Office Action issued in Canadian Appln. No. 2993551, mailed on Nov. 25, 2019, 6 pages. [cited by applicant]
Office Action issued in Canadian Appln. No. 2993551, mailed on Oct. 24, 2018, 6 pages. [cited by applicant]
Office Action issued in Russian Appin. No. 2018106248/08(009576), mailed on Oct. 23, 2018, 13 pages (with English translation). [cited by applicant]
Redding et al., “Actor-Critic Policy Learning in Cooperative Planning,” Embedded Reasoning, Papers from the 2010 AAAI Spring Symposium, Technical Report SS-10-04, Stanford, California, USA, Mar. 22-24, 2010, 7 pages. [cited by applicant]
Schulman et al., “Trust Region Policy Optimization,” CoRR, Feb. 2015, arXiv preprint arXiv:1502.05477, 16 pages. [cited by applicant]
Shibata et al., “Active Perception and Recognition Learning System Based on Actor-Q Architecture,” Systems and Computers in Japan, 2002, 11 pages. [cited by applicant]
Silver et al., “Deterministic Policy Gradient Algorithms,” Proceedings of the 31st International Conference on Machine Learning, PMLR, 2014, 32(1):387-395. [cited by applicant]
Tassa et al., “Synthesis and stabilization of complex behaviors through online trajectory optimization,” Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on. IEEE, Oct. 2012, pp. 4906-4913. [cited by applicant]
Todorov et al., “A generalized iterative LQG method for locally-optimal feedback control of constrained nonlinear stochastic systems,” American Control Conference, Proceedings of the 2005 IEEE, 2005, pp. 300-306. [cited by applicant]
Todorov et al., “MuJoCo: A physics engine for model-based control,” Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on. IEEE, Oct. 2012, pp. 5026-5033. [cited by applicant]
Uhlenbeck et al., “On the theory of the Brownian motion,” Physical Review, Sep. 1930, 36(5):823-841. [cited by applicant]
Wahlstrom et al., “From Pixels to Torques: Policy Learning with Deep Dynamical Models,” CoRR, Jun. 2015, arXiv preprint arXiv:1502.02251, 9 pages. [cited by applicant]
Watkins et al., “Q-learning,” Machine learning, May 1992, 8(3-4):279-292. [cited by applicant]
Wawrzynski et al., “Autonomous reinforcement learning with experience replay,” Neural Networks, 2013, 41:156-167. [cited by applicant]
Wawrzynski, “Real-time reinforcement learning by sequential actor-critics and experience replay,” Neural Networks, Dec. 2009, 22(10):1484-1497. [cited by applicant]
Written Opinion in Singaporean Appln. No. 11201800544U, mailed on Jan. 2, 2019, 7 pages. [cited by applicant]