IP Library › Granted Patent US 12,415,270
Granted Patent B2
US 12,415,270 · App. 17/695,753 · Granted Sep 16, 2025

Neural networks to generate robotic task demonstrations

Inventors: Ankur Handa (Seattle, WA); Iretiayo Akinola (Seattle, WA); Dieter Fox (Seattle, WA); Yashraj Shyam Narang (Seattle, WA)
Assignee: NVIDIA CORPORATION
B25J9/163B25J9/161B25J9/1612B25J9/1671
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,415,270
App. No.
17/695,753
Granted
Sep 16, 2025
Kind
B2
Abstract

A technique for training a neural network, including generating a plurality of input vectors based on a first plurality of task demonstrations associated with a first robot performing a first task in a simulated environment, wherein each input vector included in the plurality of input vectors specifies a sequence of poses of an end-effector of the first robot, and training the neural network to generate a plurality of output vectors based on the plurality of input vectors. Another technique for generating a task demonstration, including generating a simulated environment that includes a robot and at least one object, causing the robot to at least partially perform a task associated with the at least one object within the simulated environment based on a first output vector generated by a trained neural network, and recording demonstration data of the robot at least partially performing the task within the simulated environment.

Claims (41)

1. A computer-implemented method for training a neural network to enable a robot task, the computer-implemented method comprising:

generating a plurality of input vectors based on a first plurality of task demonstrations associated with a first robot performing a first task in a simulated environment, wherein each input vector included in the plurality of input vectors specifies a sequence of past poses of an end-effector of the first robot in the simulated environment; and

training the neural network to generate a plurality of output vectors based on the plurality of input vectors to generate a trained neural network, wherein the trained neural network subsequently generates training data for training a second neural network for robot learning.

2. The computer-implemented method of claim 1 , wherein training the neural network comprises:

inputting a first input vector included in the plurality of input vectors to the neural network, wherein the first input vector specifies a first sequence of poses of the end-effector; and

training the neural network to predict a first output vector based on the first input vector, wherein the first output vector comprises a second sequence of poses of the end-effector that is subsequent to the first sequence of poses of the end-effector.

3. The computer-implemented method of claim 1 , wherein a first pose of the end-effector of the first robot comprises a position and orientation of the end-effector within the simulated environment.

4. The computer-implemented method of claim 1 , wherein each input vector included in the plurality of input vectors further specifies a sequence of grip statuses of the end-effector of the first robot.

5. The computer-implemented method of claim 1 , wherein training the neural network comprises:

inputting a first input vector included in the plurality of input vectors to the neural network, wherein the first input vector specifies a first sequence of grip statuses of the end-effector; and

training the neural network to predict a first output vector based on the first input vector, wherein the first output vector comprises a second sequence of grip statuses of the end-effector that is subsequent to the first sequence of grip statuses of the end-effector.

6. The computer-implemented method of claim 1 , wherein each input vector included in the plurality of input vectors further specifies a sequence of poses associated with at least one object in the simulated environment.

7. The computer-implemented method of claim 1 , wherein training the neural network comprises:

inputting a first input vector included in the plurality of input vectors to the neural network, wherein the first input vector specifies a first sequence of poses of at least one object in the simulated environment; and

training the neural network to predict a first output vector based on the first input vector, wherein the first output vector comprises a second sequence of poses of the at least one object in the simulated environment that is subsequent to the first sequence of poses of the at least one object in the simulated environment.

8. The computer-implemented method of claim 1 , wherein none of the input vectors included in the plurality of input vectors specifies a configuration parameter that defines a model type for the first robot.

9. The computer-implemented method of claim 1 , further comprising re-training the neural network based on a second plurality of task demonstrations associated with a second robot performing a second task in a simulated environment to generate a specialized neural network, wherein the second task is different than the first task.

10. The computer-implemented method of claim 9 , wherein the first robot is defined by a first set of configuration parameters, and the second robot is defined by a second set of configuration parameters that is different than the first set of configuration parameters.

11. One or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors to train a neural network to enable a robot task by performing the steps of:

generating a plurality of input vectors based on a first plurality of task demonstrations associated with a first robot performing a first task in a simulated environment, wherein each input vector included in the plurality of input vectors specifies a sequence of past poses of an end-effector of the first robot in the simulated environment; and

training the neural network to generate a plurality of output vectors based on the plurality of input vectors to generate a trained neural network, wherein the trained neural network subsequently generates training data for training a second neural network for robot learning.

12. The one or more non-transitory computer readable media of claim 11 , wherein training the neural network comprises:

inputting a first input vector included in the plurality of input vectors to the neural network, wherein the first input vector specifies a first sequence of poses of the end-effector; and

training the neural network to predict a first output vector based on the first input vector, wherein the first output vector comprises a second sequence of poses of the end-effector that is subsequent to the first sequence of poses of the end-effector.

13. The one or more non-transitory computer readable media of claim 11 , wherein each input vector included in the plurality of input vectors comprises a state vector specifying a sequence of state poses of the end-effector and an action vector specifying a sequence of action poses of the end-effector.

14. The one or more non-transitory computer readable media of claim 11 , wherein each input vector included in the plurality of input vectors further specifies a sequence of grip statuses of the end-effector of the first robot.

15. The one or more non-transitory computer readable media of claim 11 , wherein training the neural network comprises:

inputting a first input vector included in the plurality of input vectors to the neural network, wherein the first input vector specifies a first sequence of grip statuses of the end-effector; and

training the neural network to predict a first output vector based on the first input vector, wherein the first output vector comprises a second sequence of grip statuses of the end-effector that is subsequent to the first sequence of grip statuses of the end-effector.

16. The one or more non-transitory computer readable media of claim 11 , wherein each input vector included in the plurality of input vectors further specifies a sequence of poses associated with at least one object in the simulated environment.

17. The one or more non-transitory computer readable media of claim 11 , wherein training the neural network comprises:

inputting a first input vector included in the plurality of input vectors to the neural network, wherein the first input vector specifies a first sequence of poses of at least one object in the simulated environment; and

training the neural network to predict a first output vector based on the first input vector, wherein the first output vector comprises a second sequence of poses of the at least one object in the simulated environment that is subsequent to the first sequence of poses of the at least one object in the simulated environment.

18. The one or more non-transitory computer readable media of claim 11 , further comprising:

executing the neural network to generate a second plurality of task demonstrations of a second robot performing the first task in a simulated environment.

19. The one or more non-transitory computer readable media of claim 18 , wherein the first robot is defined by a first set of configuration parameters and the second robot is defined by a second set of configuration parameters that is different than the first set of configuration parameters.

20. A system, comprising:

one or more memories that store instructions, and

one or more processors that are coupled to the one or more memories and, upon executing the instructions, train a neural network to enable a robot task by performing the steps of:

generating a plurality of input vectors based on a first plurality of task demonstrations associated with a first robot performing a first task in a simulated environment, wherein each input vector included in the plurality of input vectors specifies a sequence of past poses of an end-effector of the first robot in the simulated environment; and

train the neural network to generate a plurality of output vectors based on the plurality of input vectors to generate a trained neural network, wherein the trained neural network subsequently generates training data for training a second neural network for robot learning.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 17, 2022
From: HANDA, ANKUR; AKINOLA, IRETIAYO; FOX, DIETER; NARANG, YASHRAJ SHYAM
To: NVIDIA CORPORATION
Reel/Frame 059937/0423 →
Continuity (2)
Provisional Application 63291205 · Dec 17, 2021
Related Publication 20230191596A1 · Jun 22, 2023
References Cited (67)
US 5046022A · Conway et al. · 1991 [cited by applicant]
US 5774632A · Kaske · 1998 [cited by applicant]
US 8441454B2 · Longe et al. · 2013 [cited by applicant]
US 10387804B2 · Werth · 2019 [cited by applicant]
US 10611026B1 · Sun et al. · 2020 [cited by applicant]
US 10733502B2 · Nachum et al. · 2020 [cited by applicant]
US 10981272B1 · Nagarajan · 2021 [cited by examiner]
US 11429844B2 · Nachum et al. · 2022 [cited by applicant]
US 11904469B2 · Tsuda · 2024 [cited by examiner]
US 20080011904A1 · Cepollina et al. · 2008 [cited by applicant]
US 20090164397A1 · Kwok · 2009 [cited by applicant]
US 20090234788A1 · Kwok · 2009 [cited by applicant]
US 20120215348A1 · Skrinde · 2012 [cited by applicant]
US 20160092779A1 · Werth · 2016 [cited by applicant]
US 20170108236A1 · Guan et al. · 2017 [cited by applicant]
US 20180126553A1 · Corkum et al. · 2018 [cited by applicant]
US 20180284758A1 · Cella et al. · 2018 [cited by applicant]
US 20180345496A1 · Li · 2018 [cited by examiner]
US 20190138967A1 · Akella · 2019 [cited by applicant]
US 20190143517A1 · Yang et al. · 2019 [cited by applicant]
US 20190197396A1 · Rajkumar et al. · 2019 [cited by applicant]
US 20190202053A1 · Tan et al. · 2019 [cited by applicant]
US 20190291277A1 · Oleynic · 2019 [cited by applicant]
US 20190332922A1 · Nachum et al. · 2019 [cited by applicant]
US 20200086482A1 · Johnson et al. · 2020 [cited by applicant]
US 20200101599A1 · Yoshida · 2020 [cited by applicant]
US 20200320372A1 · Nachum et al. · 2020 [cited by applicant]
US 20200327399A1 · Silver et al. · 2020 [cited by applicant]
US 20210205988A1 · James et al. · 2021 [cited by applicant]
US 20210252698A1 · Paxton et al. · 2021 [cited by applicant]
US 20210362328A1 · Kolluri et al. · 2021 [cited by applicant]
US 20230191605A1 · Handa et al. · 2023 [cited by applicant]
US 20230330846A1 · Zhou et al. · 2023 [cited by applicant]
DE 4440859C2 · 1998 [cited by applicant]
DE 102021103272A1 · 2021 [cited by applicant]
Allshire et al., “Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World Trifinger”, arXiv:2108.09779, Aug. 22, 2021, 13 pages. [cited by applicant]
Argall et al., “A Survey of Robot Learning from Demonstration”, doi:10.1016/j.robot.2008.10.024, Robotics and Autonomous Systems, vol. 57, Nov. 25, 2009, pp. 469-483. [cited by applicant]
Zhang et al., “Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation”, IEEE, arXiv:1710.04615, Mar. 6, 2018, 9 pages. [cited by applicant]
Chebotar et al., “Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience”, International Conference on Robotics and Automation, May 20-24, 2019, pp. 8973-8979. [cited by applicant]
Chen et al., “Decision Transformer: Reinforcement Learning via Sequence Modeling”, https://sites.google.com/berkeley.edu/decision-transformer, 35th Conference on Neural Information Processing Systems, 2021, 14 pages. [cited by applicant]
Devlin et al., “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding”, arXiv:1810.04805, 2018, 16 pages. [cited by applicant]
Erickson et al., “Assistive VR Gym: Interactions with Real People to Improve Virtual Assistive Robots”, https://github.com/Healthcare-Robotics/assistive-vr-gym, IEEE, Jul. 22, 2020, pp. 299-306. [cited by applicant]
Garrett et al., “Online Replanning in Belief Space for Partially Observable Task and Motion Problems”, IEEE International Conference on Robotics and Automation, May 31-Aug. 31, 2020, pp. 5678-5684. [cited by applicant]
Gopinath et al., “Human-in-the-Loop Optimization of Shared Autonomy in Assistive Robotics”, IEEE Robotics and Automation Letters, vol. 2, No. 1, 2016, pp. 247-254. [cited by applicant]
Grice et al., “In-Home and Remote Use of Robotic Body Surrogates by People with Profound Motor Deficits”, Department of Boimedical Engineering, arXiv:1803.01477, vol. 14, No. 3, Mar. 5, 2019, 11 pages. [cited by applicant]
Ha et al., “Recurrent World Models Facilitate Policy Evolution”, 32nd Conference on Neural Information Processing Systems, arXiv:1809.01999, Sep. 4, 2018, 15 pages. [cited by applicant]
Havoutis et al., “Learning from Demonstration for Semi-Autonomous Teleoperation”, https://doi.org/10.1007/s10514-018-9745-2, Autonomous Robots, vol. 43, 2019, pp. 713-726. [cited by applicant]
Janner et al., “Offline Reinforcement Learning as One Big Sequence Modeling Problem”, 35th Conference on Neural Information Processing Systems, arXiv:2106.02039, Nov. 29, 2021, 17 pages. [cited by applicant]
Jeon et al., “Shared Autonomy with Learned Latent Actions”, arxiv:2005.03210, May 11, 2020, 10 pages. [cited by applicant]
Levine et al., “Learning Neural Network Policies with Guided Policy Search under Unknown Dynamics”, Department of Electrical Engineering and Computer Science, vol. 27, 2014, 11 pages. [cited by applicant]
Lillicrap et al., “Continuous Control with Deep Reinforcement Learning”, ICLR, arXiv:1509.02971, 2015, 14 pages. [cited by applicant]
Liu et al., “Understanding Multi-Modal Perception Using Behavioral Cloning for Peg-In-a-Hole Insertion Tasks”, arxiv:2007.11646, Jul. 22, 2020, 8 pages. [cited by applicant]
Mandlekar et al., “Roboturk: A CrowdSourcing Platform for Robotic Skill Learning Through Imitaion”, 2nd Conference on Robot Learning, PMLR, arXiv:1811.02790, Nov. 7, 2018, pp. 879-893. [cited by applicant]
Martín-Martín et al., “Variable Impedence Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks”, IEEE, arXiv: 1906.08880, Aug. 2, 2019, pp. 1010-1017. [cited by applicant]
Arpino et al., “Experimental Assessment of Human-Robot Teaming for Multi-Step Remote Manipulation with Expert Operators”, arXiv:2011, 10898, Nov. 22, 2020, 15 pages. [cited by applicant]
Arpino et al., “C-Learn: Learning Geometric Constraints from Demonstrations for Multi-Step Manipulation in Shared Autonomy”, IEEE, 2017, pp. 4058-4065. [cited by applicant]
Press et al., “Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation”, arXiv:2108.12409, Aug. 27, 2021, 23 pages. [cited by applicant]
Ratliff et al., “Riemannian Motion Policies”, arXiv:1801.02854, 2018, 15 pages. [cited by applicant]
Reddy et al., “Shared Autonomy Via Deep Reinforcement Learning”, Dept. of Electrical Engineering, arXiv: 1802.01744, May 23, 2018, 10 pages. [cited by applicant]
Csordás et al., “The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers”, arXiv:2108.12284, 2021, 16 pages. [cited by applicant]
Ross et al., “A Reduction of Imitation Learning and Structured Petition to No-Regret Online Learning”, Appearing in the Proceedings of 14th International Conference on Artificial Intelligence and Statistics, JMLR, vol. … [cited by applicant]
Schulman et al., “Proximal Policy Optimization Algorithms”, arXiv:1707.06347, Aug. 28, 2017, 12 pages. [cited by applicant]
Todorov et al., “MuJoCo: A Physics Engine for Model-Based Control”, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Oct. 7-12, 2012, pp. 5026-5033. [cited by applicant]
Vaswani et al., “Attention is All You Need”, 31st Conference on Neural Information Processing Systems, 2017, pp. 5998-6008. [cited by applicant]
Zeng et al., “Transporter Networks: Rearranging the Visual World for Robotic Manipulation”, 4th Conference on Robot Learning, arXiv:2010.14406, 2020, 23 pages. [cited by applicant]
Non Final Office Action received for U.S. Appl. No. 17/695,756 dated Apr. 24, 2024, 24 pages. [cited by applicant]
Notice of Allowance received for U.S. Appl. No. 17/695,756 dated Sep. 10, 2024, 12 pages. [cited by applicant]