IP Library › Granted Patent US 12,343,874
Granted Patent B2
US 12,343,874 · App. 18/306,711 · Granted Jul 1, 2025

Reinforcement and imitation learning for a task

Inventors: Saran Tunyasuvunakool (London, GB); Yuke Zhu (Stanford, CA); Joshua Merel (Chicago, IL); János Kramár (London, GB); Ziyu Wang (Markham, CA); Nicolas Manfred Otto Heess (London, GB)
Assignee: DeepMind Technologies Limited
B25J9/163B25J9/161B25J9/1697G06N3/008G06N3/044G06N3/045G06N3/08G06N3/084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,343,874
App. No.
18/306,711
Granted
Jul 1, 2025
Kind
B2
Abstract

A neural network control system for controlling an agent to perform a task in a real-world environment, operates based on both image data and proprioceptive data describing the configuration of the agent. The training of the control system includes both imitation learning, using datasets generated from previous performances of the task, and reinforcement learning, based on rewards calculated from control data output by the control system.

Claims (58)

1. A computer-implemented method for controlling a robotic agent to perform a task, the method comprising:

obtaining, for each of a plurality of performances of the task by a real-world agent controlled by an operator in a real-world environment, a respective demonstration dataset characterizing the corresponding performance of the task in the real-world environment; and

training a neural network for controlling a simulated robotic agent to perform the task in a simulated environment using the demonstration dataset, the training comprising:

obtaining (i) simulated image data encoding simulated camera images characterizing a current state of the simulated environment and (ii) simulated proprioceptive data comprising one or more variables characterizing configurations of the simulated robotic agent;

processing at least (i) the simulated image data and (ii) the simulated proprioceptive data using the neural network, according to current values of parameters of the neural network, to generate one or more sets of control commands for controlling movements of a plurality of components of the simulated robotic agent;

for each set of control commands, computing a task reward value characterizing how successfully the task is carried out upon implementation of the set of control commands by the simulated robotic agent in the simulated environment; and

adjusting the parameters of the neural network based on a hybrid energy function including (i) an imitation reward value derived using the demonstration datasets obtained for the real-world environment and the sets of control commands generated for the simulated environment and (ii) a task reward term computed using the task reward values; and

using the trained neural network to control the real-world robotic agent to perform the task in the real-world environment.

2. The method of the claim 1 , wherein the training further comprises:

using the demonstration datasets to generate a discriminator network; and

determining the imitation reward value using the discriminator network and the sets of one or more control commands.

3. The method of claim 2 , wherein the discriminator network is configured to receive data characterizing positions of one or more objects in the simulated environment.

4. The method of claim 1 , wherein generating the task reward value comprises:

simulating a process carried out by the simulated robotic agent in the simulated environment based on the corresponding set of control commands to generate a final state of the simulated environment; and

computing an initial task reward value based at least on the final state of the simulated environment.

5. The method of claim 4 , wherein updates to the neural network are calculated using an activation function estimator obtained by subtracting a value function from the initial task reward value, and the initial reward value is calculated according to a task reward function based on the final state of the environment.

6. The method of claim 5 , wherein the value function is computed using data characterizing positions of one of more objects in the simulated environment.

7. The method of claim 5 , wherein the value function is computed by an adaptive model.

8. The method of claim 1 , wherein the neural network comprises:

a convolutional neural network configured to process the simulated image data to generate convolved data; and

at least one adaptive component configured to process the output of the convolutional neural network and the simulated proprioceptive data.

9. The method according to claim 8 , wherein the adaptive component is a perceptron.

10. The method of claim 8 , wherein the neural network further comprises a recursive neural network configured to receive input data generated both from the simulated image data and the simulated proprioceptive data.

11. The method of claim 8 , further comprising:

defining at least one auxiliary task, and training the convolutional network as part of an adaptive system which is trained to perform the auxiliary task based on the simulated image data.

12. The method of claim 1 , wherein:

the training of the neural network is performed in parallel with the training of a plurality of additional instances of the neural network by respective workers.

13. The method of claim 1 , wherein using the neural network to generate the control commands comprises:

defining a plurality of task stages of the task, wherein each respective task stage defines a corresponding portion in a plurality of portions of the task;

for each respective task stage, generating a respective set of demonstration states, wherein the respective set of demonstration states are selected from states defined in one or more of the demonstration datasets and characterize performance of the portion of task corresponding to the respective task stage;

randomly selecting, with a predefined probability, a task stage from the plurality of task stages;

randomly selecting an initial state from the respective set of demonstration states generated for the selected task stage; and

providing, as input to the neural network, data characterizing the simulated environment and the simulated agent at the respective initial stage.

14. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for controlling a robotic agent to perform a task, the operations comprising:

obtaining, for each of a plurality of performances of the task by a real-world agent controlled by an operator in a real-world environment, a respective demonstration dataset characterizing the corresponding performance of the task in the real-world environment; and

training a neural network for controlling a simulated robotic agent to perform the task in a simulated environment using the demonstration dataset, the training comprising:

obtaining (i) simulated image data encoding simulated camera images characterizing a current state of the simulated environment and (ii) simulated proprioceptive data comprising one or more variables characterizing configurations of the simulated robotic agent;

processing at least (i) the simulated image data and (ii) the simulated proprioceptive data using the neural network, according to current values of parameters of the neural network, to generate one or more sets of control commands for controlling movements of a plurality of components of the simulated robotic agent;

for each set of control commands, computing a task reward value characterizing how successfully the task is carried out upon implementation of the set of control commands by the simulated robotic agent in the simulated environment; and

adjusting the parameters of the neural network based on a hybrid energy function including (i) an imitation reward value derived using the demonstration datasets obtained for the real-world environment and the sets of control commands generated for the simulated environment and (ii) a task reward term computed using the task reward values; and

using the trained neural network to control the real-world robotic agent to perform the task in the real-world environment.

15. The system of claim 14 , wherein the training further comprises:

using the demonstration datasets to generate a discriminator network; and

determining the imitation reward value using the discriminator network and the sets of one or more control commands.

16. The system of claim 15 , wherein the discriminator network is configured to receive data characterizing positions of one or more objects in the simulated environment.

17. The system of claim 14 , wherein generating the task reward value comprises:

simulating a process carried out by the simulated robotic agent in the simulated environment based on the corresponding set of control commands to generate a final state of the simulated environment; and

computing an initial task reward value based at least on the final state of the simulated environment.

18. The system of claim 17 , wherein updates to the neural network are calculated using an activation function estimator obtained by subtracting a value function from the initial task reward value, and the initial reward value is calculated according to a task reward function based on the final state of the environment.

19. The system of claim 18 , wherein the value function is computed using data characterizing positions of one of more objects in the simulated environment.

20. One or more non transitory computer storage media storing computer program instructions that, when executed by one or more computers, cause the one or more computers to perform operations for controlling a robotic agent to perform a task, the operations comprising:

obtaining, for each of a plurality of performances of the task by a real-world agent controlled by an operator in a real-world environment, a respective demonstration dataset characterizing the corresponding performance of the task in the real-world environment; and

training a neural network for controlling a simulated robotic agent to perform the task in a simulated environment using the demonstration dataset, the training comprising:

obtaining (i) simulated image data encoding simulated camera images characterizing a current state of the simulated environment and (ii) simulated proprioceptive data comprising one or more variables characterizing configurations of the simulated robotic agent;

processing at least (i) the simulated image data and (ii) the simulated proprioceptive data using the neural network, according to current values of parameters of the neural network, to generate one or more sets of control commands for controlling movements of a plurality of components of the simulated robotic agent;

for each set of control commands, computing a task reward value characterizing how successfully the task is carried out upon implementation of the set of control commands by the simulated robotic agent in the simulated environment; and

adjusting the parameters of the neural network based on a hybrid energy function including (i) an imitation reward value derived using the demonstration datasets obtained for the real-world environment and the sets of control commands generated for the simulated environment and (ii) a task reward term computed using the task reward values; and

using the trained neural network to control the real-world robotic agent to perform the task in the real-world environment.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 6, 2025
From: DEEPMIND TECHNOLOGIES LIMITED
To: GDM HOLDING LLC
Reel/Frame 071498/0210 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 10, 2023
From: TUNYASUVUNAKOOL, SARAN; ZHU, YUKE; MEREL, JOSHUA; KRAMAR, JANOS; WANG, ZIYU; HEESS, NICOLAS MANFRED OTTO
To: DEEPMIND TECHNOLOGIES LIMITED
Reel/Frame 063599/0449 →
Continuity (3)
Continuation 16174112 · Oct 29, 2018
Provisional Application 62578368 · Oct 27, 2017
Related Publication 20230330848A1 · Oct 19, 2023
References Cited (73)
US 10926408B1 · Vogelsong · 2021 [cited by examiner]
US 11341364B2 · Bousmalis · 2022 [cited by examiner]
US 11651208B2 · Xu · 2023 [cited by examiner]
US 20170024643A1 · Lillicrap et al. · 2017 [cited by applicant]
US 20170032035A1 · Gao et al. · 2017 [cited by applicant]
US 20180032863A1 · Graepel · 2018 [cited by examiner]
US 20180165603A1 · Van Seijen et al. · 2018 [cited by applicant]
US 20190232489A1 · Pascanu · 2019 [cited by examiner]
US 20230256593A1 · Zolna · 2023 [cited by examiner]
CN 105637540 · 2016 [cited by applicant]
Reinforcement and Imitation Learning for Diverse Visumotor Skills, Under review as a conference paper at ICLR 2018. [cited by applicant]
Boularias et al. “Relative entropy inverse reinforcement learning,” Proceedings of the International Conference on Artificial Intelligence and Statistics, Jun. 2011, 8 pages. [cited by applicant]
Bousmalis et al. “Using Simulation and Domain Adaptation to improved Efficiency of Deep Robotic Grasping,” arXiv 1709.07857v2, Sep. 25, 2017, 9 pages. [cited by applicant]
Duan et al. “One-shot imitation learning,” arXiv 1703.07326v3, Dec. 4, 2017, 27 pages. [cited by applicant]
Edmonds et al., “Feeling the force: Integrating force and pose for fluent discovery through imitation learning to open medicine bottles,” 2017 EIII/RSJ International Conference and Intelligent Robots and Systems, Sep. 2… [cited by applicant]
EP Office Action in European Appln. No. 18203124.5, dated Feb. 18, 2021, 15 pages. [cited by applicant]
Extended European Search Report in European Appln. 18203124.5-1221, dated Apr. 8, 2019, 14 pages. [cited by applicant]
Finn et al. “Guided cost learning: Deep inverse optimal control via policy optimization,” Proceedings of the 33 [cited by applicant]
Finn et al. “One-shot Visual Imitation Learning via Meta-Learning,” arXiv 1709.04905v1, Sep. 14, 2017, 12 pages. [cited by applicant]
Goodfellow et al. “Generative adversarial nets,” NIPS Dec. 2014, 9 pages. [cited by applicant]
Gu et al. “Continuous deep Q-learning with model-based acceleration,” ICML, Jun. 2016, 10 pages. [cited by applicant]
Gu et al. “Deep reinforcement learning for robotic manipulation,” arXiv 1610.00633v1, Oct. 3, 2016, 9 pages. [cited by applicant]
Gupta et al. “Learning dexterous manipulation for a soft robotic hand from human demonstrations,” arXiv 1603.06348v3, Mar. 20, 2017, 9 pages. [cited by applicant]
Heess et al. “Learning and transfer of modulated locomotor controllers,” arXiv 1610.05182, Oct. 17, 2016, 13 pages. [cited by applicant]
Heess et al. “Learning continuous control policies by stochastic value gradients,” NIPS, Dec. 2015, 9 pages. [cited by applicant]
HM Biu et al. Using grayscale images for object recognition with convolutional-recursive neural network. Jul. 2016. [retrieved from internet on Nov. 19, 2022] <U RL: https://ieeexplore.ieee.org/abstract/document/7562656… [cited by applicant]
Ho et al. “Generative adversarial imitation learning,” NIPS, Dec. 2016, 9 pages. [cited by applicant]
Jaderberg et al. “Reinforcement learning with unsupervised auxiliary tasks,” arXiv 1611.05397, Nov. 16, 2016, 14 pages. [cited by applicant]
James et al. “Transferring end-to-end visuomotor control from simulation to real world for a multi-stage task,” arXiv 1707.02267v2, Oct. 17, 2017, 10 pages. [cited by applicant]
Kakade et al. “Approximately optimal approximate reinforcement learning,” ICML Jul. 2002, 8 pages. [cited by applicant]
Kingma et al. “Adam: A method of stochastic optimization,” arXiv 1412.6980v9, Jan. 30, 2017, 15 pages. [cited by applicant]
Kroemer et al., “Combining active learning and reactive control for robot grasping,” Robotics and Autonomous Systems, Sep. 2010, 58(9):1105-1116. [cited by applicant]
Kumar et al. “Learning dexterous manipulation policies from experience and imitation,” arXiv 1611.05095, Nov. 15, 2016, 17 pages. [cited by applicant]
Levine et al. “End-to-end training of deep visuomotor policies,” Journal of Machine Learning Research, 17(1), Jan. 2016, 40 pages. [cited by applicant]
Levine et al. “Guided policy search,” ICML, Feb. 2013, 9 pages. [cited by applicant]
Levine et al. “Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection,” arXiv 1603.02199v4, Aug. 28, 2016, 12 pages. [cited by applicant]
Li et al. “InfoGAIL: Interpretabe Imitation Learning from Visual Demonstrations,” arXiv 1703.08840v2, Nov. 14, 2017, 14 pages. [cited by applicant]
Li et al. Inferring The Latent Structure of Human Decision-Making from Raw Visual Inputs. arXiv. Mar. 26, 2017. arXiv:1703.08840 (Year: 2017). [cited by applicant]
Lillicrap et al. “Continuous control with deep reinforcement learning,” arXiv 1509.02971v5, Feb. 29, 2016, 14 pages. [cited by applicant]
Liu et al. “Imitation from observation: Learning to imitate behaviors from raw video via context translation,” arXiv 1707.03374v2, Jun. 18, 2018, 8 pages. [cited by applicant]
Merel et al. “Learning human behaviors from motion capture by adversarial imitation,” arXiv 1707.02201v2 Jul. 10, 2017, 12 pages. [cited by applicant]
Mnih et al. “Human-level control through deep reinforcement learning,” Nature 518(7540) Feb. 2015, 13 pages. [cited by applicant]
Nair et al. “Overcoming exploration in reinforcement learning with demonstrations,” arXiv 1709.10089v2, Feb. 25, 2018, 8 pages. [cited by applicant]
Nair et al. Massively Parallel Methods for Deep Reinforcement Learning. arXiv. Jul. 1, 2015. arXiv:1507.04296 (Year: 2015). [cited by applicant]
Ng et al. “Policy invariance under reward transformation: Theory and Application to reward shaping,” ICML, Jun. 1999, 10 pages. [cited by applicant]
Office Action in Chinese Appln. No. 201811269614.4, dated Sep. 22, 2022, 19 pages (with English translation). [cited by applicant]
Peng et al. “Sim-to-Real Transfer of Robotic Control with Dynamics Randomization,” arXiv 1710.06537v3, Mar. 3, 2018, 8 pages. [cited by applicant]
Pinto et al. “Asymmetric Actor Critic for Image-Based Robot Learning,” arXiv 1710.06542v1, Oct. 18, 2017, 8 pages. [cited by applicant]
Pinto et al. “Supervising self-supervision: Learning to grasp from 50k tries and 700 robot hours,” arXiv 1509.06825, Sep. 23, 2015, 8 pages. [cited by applicant]
Popov et al. “Data-efficient deep reinforcement learning for dexterous manipulation,” arXiv 1704.03073, Apr. 10, 2017, 12 pages. [cited by applicant]
Rahmatizadeh et al. “Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration,” arXiv 1707.02920v2, Apr. 22, 2018, 8 pages. [cited by applicant]
Rajeswaran et al. “Learning complex dexterous manipulation with deep reinforcement learning and demonstrations,” arXiv 1709.10087v2, Jun. 26, 2018, 9 pages. [cited by applicant]
Ross et al. “A reduction of imitation learning and structured prediction to no-regret online learning,” ICAIS Jun. 2011, 9 pages. [cited by applicant]
Rusu et al. “Progressive neural networks,” arXiv 1606.04671v3 Sep. 7, 2016, 14 pages. [cited by applicant]
Rusu et al. “Sim-to-real robot learning from pixels with progressive nets,” arXiv 1610.04286v2, May 22, 2018, 9 pages. [cited by applicant]
S James and E Johns. 3D Simulation for Robot Arm Control with Deep 0-Learning. arXiv. Dec. 13, 2016. arXiv:1609.03759 (Year: 2016). [cited by applicant]
Schenck et al. “Reasoning about liquids via closed-loop simulation,” arXiv 1703.01656v2 Jun. 9, 2017, 10 pages. [cited by applicant]
Schulman et al. “Proximal policy optimization algorithms,” arXiv 1707.06347v2, Aug. 28, 2017, 12 pages. [cited by applicant]
Schulman et al. “Trust region policy optimization,” International Conference on Machine Learning, Jun. 2015, 9 pages. [cited by applicant]
Sermanet et al. “Time-constrastive networks: Self-supervised learning from multi-view observation,” arXiv 1704.06888v3, Mar. 20, 2018, 15 pages. [cited by applicant]
Silver et al. “Deterministic policy gradient algorithms,” ICML, Jun. 2014, 9 pages. [cited by applicant]
Silver et al. “Mastering the game of go with deep neural networks and tree search,” Nature 529(7587) Jan. 2016, 20 pages. [cited by applicant]
Singh et al., “GLPAC: Generalizing vision-based robotic skills using weakly labeled images,” arXiv, Aug. 7, 2017, 12 pages. [cited by applicant]
Singh, S.P. Transfer of learning by composing solutions of elemental sequential tasks. Mach Learn 8, 323-339 (1992). https://doi.org/10.1007/BF00992700 (Year: 1992). [cited by applicant]
Tobin et al. “Domain randomization for transferring deep neural networks from simulation to the real world,” arXiv 1703.06907v1, Mar. 20, 2017, 8 pages. [cited by applicant]
Todorov et al. “Mujoco: A physics engine for model-based control,” IEEE/RSJ International Conference on Intelligent Robots and Systems, Oct. 2012, 8 pages. [cited by applicant]
Vecerik et al. “Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards,” arXiv 1707.08817v1, Jul. 27, 2017, 10 pages. [cited by applicant]
Viereck et al. “Learning a visumotor controller for real world robotic grasping using easily simulated depth images,” arXiv 1706.04652v3, Nov. 17, 2017, 10 pages. [cited by applicant]
Wang et al. “Robust imitation of diverse behaviors,” NIPS, Dec. 2017, 10 pages. [cited by applicant]
Wulfmeier et al, “Addressing appearance change in outdoor robotics with adversarial domain adaptation,” arXiv, Sep. 17, 2007, 8 pages. [cited by applicant]
Yahya et al. “Collective robot reinforcement learning with distributed asynchronous guided policy search,” arXiv 1610.00673, Oct. 3, 2016, 8 pages. [cited by applicant]
Zeng et al., “Visual Forecasting by Imitating Dynamics in Natural Sequences,” 2017 IEEE International Conference on Computer Vision, Oct. 2017, pp. 3018-3027. [cited by applicant]
Zhu et al, “Reinforcement and imitation learning for diverse visuomotor skills,” arXiv, May 27, 2018, 12 pages. [cited by applicant]