IP Library › Granted Patent US 12,585,917
Granted Patent B2
US 12,585,917 · App. 17/704,721 · Granted Mar 24, 2026

Reinforcement learning using advantage estimates

Inventors: Shixiang Gu (Cambridge, GB); Timothy Paul Lillicrap (London, GB); Ilya Sutskever (San Francisco, CA); Sergey Vladimir Levine (Berkeley, CA)
Assignee: Google LLC
G06N3/042G06N3/092
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,585,917
App. No.
17/704,721
Granted
Mar 24, 2026
Kind
B2
Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for computing Q values for actions to be performed by an agent interacting with an environment from a continuous action space of actions. In one aspect, a system includes a value subnetwork configured to receive an observation characterizing a current state of the environment and process the observation to generate a value estimate; a policy subnetwork configured to receive the observation and process the observation to generate an ideal point in the continuous action space; and a subsystem configured to receive a particular point in the continuous action space representing a particular action; generate an advantage estimate for the particular action; and generate a Q value for the particular action that is an estimate of an expected return resulting from the agent performing the particular action when the environment is in the current state.

Claims (45)

1 . A method for controlling a robot interacting with a real-world environment, the method comprising:

receiving an observation characterizing a current state of the environment; and

processing the observation using a policy subnetwork to generate an output action from a set of actions that lie on a continuous domain, wherein:

the output action comprises a plurality of control inputs for the robot;

the policy subnetwork has been trained jointly with a value subnetwork on a set of training experience tuples using an error function,

the value subnetwork is configured to process an input observation characterizing an input state to generate as output a value estimate for the input state, the value estimate being an estimate of an expected return resulting from the environment being in the input state irrespective of which action is performed in response to the input observation,

each training experience tuple identifies i) a training observation characterizing a training state of a training environment, ii) a training action performed by an agent in response to the training observation, iii) a reward received as a result of the agent performing the training action in response to the training observation, and iv) a subsequent observation characterizing a subsequent state of the environment, and

the error function for the joint training of the value subnetwork and the policy subnetwork measures, for each training experience tuple, an error between i) a Q value for the training action identified in the training observation that is generated using an advantage estimate for the training action that is based on a distance between a) an output action generated by processing the training observation using the policy subnetwork and b) the training action identified in the training observation and ii) a target Q value that is generated based on c) the reward identified in the training experience tuple and d) a value estimate generated by processing the subsequent observation identified in the training experience tuple using the value subnetwork; and

causing the robot to perform the output action by applying the plurality of control inputs to the robot.

2 . The method of claim 1 , wherein the agent is the robot and the training environment is the real-world environment.

3 . The method of claim 1 , wherein the agent is a simulated agent and the training environment is a computer simulation of the real-world environment.

4 . The method of claim 1 , wherein the Q value for the training action is based on a value estimate generated by processing the training observation identified in the training experience tuple using the value subnetwork and the distance between a) the output action generated by processing the training observation using the policy subnetwork and b) the training action identified in the training observation.

5 . The method of claim 1 , wherein the advantage estimate is computed based on applying a function having state-dependent parameters to the distance between a) the output action generated by processing the training observation using the policy subnetwork and b) the training action identified in the training observation.

6 . The method of claim 5 , wherein the state-dependent parameters are generated by processing, by a function parameter neural network the training observation to generate an output that defines values of the state-dependent parameters.

7 . The method of claim 1 , wherein the policy subnetwork and the value subnetwork are also jointly trained on a set of imagination rollouts, wherein each imagination rollout is a synthetic experience tuple.

8 . The method of claim 1 , wherein the policy subnetwork has been trained by backpropagating gradients of the error function to update values of a plurality of parameters of the policy subnetwork.

9 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for controlling a robot interacting with a real-world environment, the operations comprising:

receiving an observation characterizing a current state of the environment; and

processing the observation using a policy subnetwork to generate an output action from a set of actions that lie on a continuous domain, wherein:

the output action comprises a plurality of control inputs for the robot;

the policy subnetwork has been trained jointly with a value subnetwork on a set of training experience tuples using an error function,

the value subnetwork is configured to process an input observation characterizing an input state to generate as output a value estimate for the input state, the value estimate being an estimate of an expected return resulting from the environment being in the input state irrespective of which action is performed in response to the input observation,

each training experience tuple identifies i) a training observation characterizing a training state of a training environment, ii) a training action performed by an agent in response to the training observation, iii) a reward received as a result of the agent performing the training action in response to the training observation, and iv) a subsequent observation characterizing a subsequent state of the environment, and

the error function for the joint training of the value subnetwork and the policy subnetwork measures, for each training experience tuple, an error between i) a Q value for the training action identified in the training observation that is generated using an advantage estimate for the training action that is based on a distance between a) an output action generated by processing the training observation using the policy subnetwork and b) the training action identified in the training observation and ii) a target Q value that is generated based on c) the reward identified in the training experience tuple and d) a value estimate generated by processing the subsequent observation identified in the training experience tuple using the value subnetwork; and

causing the robot to perform the output action by providing the plurality of control inputs to the robot.

10 . The system of claim 9 , wherein the agent is the robot and the training environment is the real-world environment.

11 . The system of claim 9 , wherein the agent is a simulated agent and the training environment is a computer simulation of the real-world environment.

12 . The system of claim 9 , wherein the Q value for the training action is based on a value estimate generated by processing the training observation identified in the training experience tuple using the value subnetwork and the distance between a) the output action generated by processing the training observation using the policy subnetwork and b) the training action identified in the training observation.

13 . The system of claim 9 , wherein the advantage estimate is computed based on applying a function having state-dependent parameters to the distance between a) the output action generated by processing the training observation using the policy subnetwork and b) the training action identified in the training observation.

14 . The system of claim 13 , wherein the state-dependent parameters are generated by processing, by a function parameter neural network the training observation to generate an output that defines values of the state-dependent parameters.

15 . The system of claim 9 , wherein the policy subnetwork and the value subnetwork are also jointly trained on a set of imagination rollouts, wherein each imagination rollout is a synthetic experience tuple.

16 . The system of claim 9 , wherein the policy subnetwork has been trained by backpropagating gradients of the error function to update values of a plurality of parameters of the policy subnetwork.

17 . One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for controlling a robot interacting with a real-world environment, the operations comprising:

receiving an observation characterizing a current state of the environment; and

processing the observation using a policy subnetwork to generate an output action from a set of actions that lie on a continuous domain, wherein:

the output action comprises a plurality of control inputs for the robot;

the policy subnetwork has been trained jointly with a value subnetwork on a set of training experience tuples using an error function,

the value subnetwork is configured to process an input observation characterizing an input state to generate as output a value estimate for the input state, the value estimate being an estimate of an expected return resulting from the environment being in the input state irrespective of which action is performed in response to the input observation,

each training experience tuple identifies i) a training observation characterizing a training state of a training environment, ii) a training action performed by an agent in response to the training observation, iii) a reward received as a result of the agent performing the training action in response to the training observation, and iv) a subsequent observation characterizing a subsequent state of the environment, and

the error function for the joint training of the value subnetwork and the policy subnetwork measures, for each training experience tuple, an error between i) a Q value for the training action identified in the training observation that is generated using an advantage estimate for the training action that is based on a distance between a) an output action generated by processing the training observation using the policy subnetwork and b) the training action identified in the training observation and ii) a target Q value that is generated based on c) the reward identified in the training experience tuple and d) a value estimate generated by processing the subsequent observation identified in the training experience tuple using the value subnetwork; and

causing the robot to perform the output action by providing the plurality of control inputs to the robot.

18 . The computer-readable media of claim 17 , wherein the agent is the robot and the training environment is the real-world environment.

19 . The computer-readable media of claim 17 , wherein the agent is a simulated agent and the training environment is a computer simulation of the real-world environment.

20 . The computer-readable media of claim 17 , wherein the Q value for the training action is based on a value estimate generated by processing the training observation identified in the training experience tuple using the value subnetwork and the distance between a) the output action generated by processing the training observation using the policy subnetwork and b) the training action identified in the training observation.

21 . The method of claim 1 , wherein the plurality of control inputs comprise a respective control input for each of one or more joints of the robot.

Assignments (2)
CHANGE OF NAME Recorded Mar 31, 2022
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 059567/0048 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 31, 2022
From: GU, SHIXIANG; LILLICRAP, TIMOTHY PAUL; SUTSKEVER, ILYA; LEVINE, SERGEY VLADIMIR
To: GOOGLE INC.
Reel/Frame 059458/0764 →
Continuity (3)
Continuation 15429088 · Feb 9, 2017
Provisional Application 62293250 · Feb 9, 2016
Related Publication 20220284266A1 · Sep 8, 2022
References Cited (73)
US 9189749B2 · Estes · 2015 [cited by applicant]
US 9536191B1 · Arel · 2017 [cited by examiner]
US 11288568B2 · Gu · 2022 [cited by examiner]
US 20050245303A1 · Graepel · 2005 [cited by applicant]
US 20090098515A1 · Das et al. · 2009 [cited by applicant]
US 20090099985A1 · Tesauro et al. · 2009 [cited by applicant]
US 20100094788A1 · Schafer · 2010 [cited by examiner]
US 20130185039A1 · Tesauro et al. · 2013 [cited by applicant]
US 20130218814A1 · Kang · 2013 [cited by examiner]
US 20150301510A1 · Düll · 2015 [cited by examiner]
US 20170213150A1 · Arel · 2017 [cited by examiner]
CN 103282891 · 2013 [cited by applicant]
CN 103473111 · 2013 [cited by applicant]
CN 105139072 · 2015 [cited by applicant]
EP 2386987 · 2013 [cited by applicant]
JP 2010134907 · 2010 [cited by applicant]
Fonteneau et al., “Batch Mode Reinforcement Learning based on the Synthesis of Artificial Trajectories,” Springer Science+Business Media New York (2012) (Year: 2012). [cited by examiner]
Faust et al., “Continuous Action Reinforcement Learning for Control-Affine Systems with Unknown Dynamics,” UNM (2014) (Year: 2014). [cited by examiner]
Gu et al., “Continuous Deep Q-Learning with Model-based Acceleration,” arXiv (2016) (Year: 2016). [cited by examiner]
Gu et al., “Towards Deep Neural Network Architectures Robust to Adversarial Examples,” arXiv (2014) (Year: 2014). [cited by examiner]
Atkeson et al. “Locally weighted learning for control,” Lazy learning, Springer Netherlands, 1997, 63 pages. [cited by applicant]
Baird III. “Advantage updating,” No. WL-TR-93-1146. Wright Lab Wright-Patterson AFB, OH, Nov. 4, 1993. [cited by applicant]
CN Office Action in Chinese Application No. 201780015574.8, dated May 25, 2021, 33 pages (with English translation). [cited by applicant]
De Buin et al. “The importance of experience replay database composition in deep reinforcement learning,” Deep Reinforcement Learning Workshop, NIPS, 2015, 9 pages. [cited by applicant]
Deisenroth et al. “A survey on policy search for robotics,” Foundations and Trends® in Robotics 2.1-2, Aug. 30, 2013, 143 pages. [cited by applicant]
Deisenroth et al. “PILCO: A model-based and data-efficient approach to policy search,” Proceedings of the 28th International Conference on machine learning, 2011, 8 pages. [cited by applicant]
EP Office Action in European Application No. 17714039.9, dated Aug. 21, 2020, 15 pages. [cited by applicant]
Fonteneau et al., “Batch Mode Reinforcement Learning based on the Synthesis of Artificial Trajectories,” Spring Science+Business Media New York 2012. [cited by applicant]
Fonteneau et al., “Simultaneous Perturbation Algorithms for Batch Off-Policy Search,” (IEEE 2014) (Year: 2014). [cited by applicant]
Fu et al. “One-shot learning of manipulation skills with online dynamics adaptation and neural etwork priors,” arXiv preprint arXiv1509.06841v3, Aug. 11, 2016, 8 pages. [cited by applicant]
Gu et al. “Continuous Deep Q-learning with Model-based Acceleration,” arXiv preprint arXiv1603.00748, Mar. 2, 2016, 13 pages. [cited by applicant]
Hafner et al. “Reinforcement learning in feedback control,” Machine learning 84.1, Jul. 1, 2011, 33 pages. [cited by applicant]
Harmon et al. “Multi-player residual advantage learning with general function approximation,” Wright Laboratory, WL/AACF, Wright-Patterson Air Force Base, OH, 1996, 14 pages. [cited by applicant]
Hasselt et al., “Using Continuous Action Spaces to Solve Discrete Problems” IEEE (2009) (Year: 2009). [cited by applicant]
Hausknecht et al. “Deep reinforcement learning in parameterized action space,” arXiv preprint arXiv 1511.04143v4, Feb. 16, 2016, 12 pages. [cited by applicant]
Hausknecht et al., “Deep Reinforcement Learning in Parnneterized Action Space,” ICLR 2016 (Year: 2016). [cited by applicant]
Heess et al. “Learning continuous control policies by stochastic value gradients,” Advances in Neural Information Processing Systems, 2015, 9 pages. [cited by applicant]
International Search Report and Written Opinion issued in international application No. PCT/US2017/017258, mailed on Jun. 19, 2017, 22 pages. [cited by applicant]
JP Decision to Grant a Patent in Japanese Appln. No. 2018-560745, dated Feb. 3, 2020, 5 pages (with English translation). [cited by applicant]
JP Office Action in Japanese Appln No. 2018-560745, dated Oct. 15, 2019, 7 pages (with English translation). [cited by applicant]
Kingma et al. “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980v9 Jan. 30, 2017, 15 pages. [cited by applicant]
Kober et al. “Reinforcement learning in robotics: A survey,” The International Journal of Robotics Rsearch 32(11), Sep. 2013, 74 pages. [cited by applicant]
Konda et al. “Actor-critic algorithms,” NIPS. vol. 13, Nov. 1999, 7 pages. [cited by applicant]
Langley. “Crafting papers on Machine Learning,” Proceedings of the 17th International Conference on Machine Learning, vol. 34, 2000, 7 pages. [cited by applicant]
Levine et al. “End-to-end training of deep visuomotor policies,” Journal of Machine Learning Research, 17(39), Jan. 1, 2016, 40 pages. [cited by applicant]
Levine et al. “Guided policy search,” ICML (3), 2013, 9 pages. [cited by applicant]
Levine et al. “Learning neural network policies with guided policy search under unknown dynamics,” Advances in Neural Information Processing Systems, 2014, 9 pages. [cited by applicant]
Li et al. “Iterative linear quadratic regulator design for nonlinear biological movement systems,” Submitted to the 1st International Conference on Informatics in Control, Automation and Robotics, Aug. 2004, 8 pages. [cited by applicant]
Lillicrap et al. “Continuous control with deep reinforcement learning,” arXiv preprint arXiv1509.02791v4, Jan. 19, 2016, 14 pages. [cited by applicant]
Lillicrap et al. “Continuous control with deep reinforcement learning,” arXiv preprint arXiv1509.02971v5, Feb. 29, 2016, 14 pages. [cited by applicant]
Mnih et al. “Human-level control through deep reinforcement learning,” Nature 518.7540, Feb. 26, 2015, 13 pages. [cited by applicant]
Mnih et al. “Playing Atari with deep reinforcement learning,” arXiv preprint arXiv1312.5602, Dec. 19, 2013, 9 pages. [cited by applicant]
Oh et al. “Action-conditional video prediction using deep networks in Atari games,” Advances in Neural Information Processing Systems, 2015, 9 pages. [cited by applicant]
Parisotto et al., “Actor-Mimic Deep Multitask and Transfer Reinforcement Learning,” ICLR 2016 (Jan. 6, 2016) (Year: 2016). [cited by applicant]
Peters et al. “Policy gradient methods for robotics,” International Conference on Intelligent Robots and Systems, Oct. 2006, 7 pages. [cited by applicant]
Peters et al. “Relative entropy policy search,” AAAI, Jul. 11, 2010, 6 pages. [cited by applicant]
Rawlik et al. “On stochastic optimal control and reinforcement learning by approximate inference,” Proceedings of Robotics; Science and Systems VIII, Jul. 2012, 16 pages. [cited by applicant]
Schaul et al. “Prioritized experience replay,” arXiv preprint arXiv:1511.05952v4, Feb. 25, 2016, 21 pages. [cited by applicant]
Schulman et al. “High-dimensional continuous control using generalized advantage estimation,” arXiv preprint arXiv1506.02438v5, Sep. 9, 2016, 14 pages. [cited by applicant]
Schulman et al. “Trust region policy Optimization,” International Conference on Machine Learning, Feb. 19, 2015, 9 pages. [cited by applicant]
Silver et al. “Deterministic policy gradient algorithms,” International Conference on Machine Learning, Jun. 21, 2014, 9 pages. [cited by applicant]
Sutton et al. “Policy gradient methods for reinforcement learning with function approximation,” NIPS (99), Nov. 29, 1999, 7 pages. [cited by applicant]
Sutton. “Dyna, an integrated architecture for learning, planning, and reacting,” SIGART Newsletter, Association for Computing Machinery, New York, NY, 2(4) Jul. 1, 1991, 4 pages. [cited by applicant]
Sutton. “Integrated architectures for learning, planning, and reacting based on approximating dynamic programming,” Proceedings of the 7th International Conference on Machine Learning, 1990, 9 pages. [cited by applicant]
Tassa et al. “Synthesis and stabilization of complex behaviors through online trajectory optimization,” International Conference on Intelligent Robots and Systems, Oct. 7, 2012, 8 pages. [cited by applicant]
Todorov et al. “Mujoco: A physics engine for model-based control,” International Conference on Intelligent Robots and Systems, Oct. 7, 2012, 8 pages. [cited by applicant]
Wahlstrom et al. “From pixels to torques: Policy learning with deep dynamical models,” arXiv preprint arXiv:1502.02251v3, Jun. 18, 2015, 9 pages. [cited by applicant]
Wang et al. “Dueling network architectures for deep reinforcement learning,” arXiv preprint arXiv:1511.06581v2, , Jan. 8, 2016, 16 pages. [cited by applicant]
Wang et al. “Dueling network architectures for deep reinforcement learning,” arXiv preprint arXiv:1511.06581v3, Apr. 5, 2016, 15 pages. [cited by applicant]
Watter et al. “Embed to control: A locally linear latent dynamics model for control from raw images,” Advances in Neural Information Processing Systems, 2015, 9 pages. [cited by applicant]
Yang et al., “Control of Nonaffine Nonlinear Discrete-Time Systems using Reinforcement-Learning-Based Linearly Parametrized Neural Networks,” IEEE (2008) (Year: 2008). [cited by applicant]
Yegor Tkachenko, “Autonomous CRM Control via CLV Approximation with Deep Reinforcement Learning in Discrete and Continuous Action Space” Stanford University (2015) (Year: 2015). [cited by applicant]
Yu, “Playing Othello by Deep Learning Neural Network,” Final Year Project Detailed Intermediate Report, 2013, 8 pages. [cited by applicant]