Simulated local demonstration data for robotic demonstration learning

1. A computer-implemented method comprising:

receiving perceptual data of a workcell at a first site, the workcell having a physical robot to be configured to execute a task according to a skill template, wherein the skill template specifies a plurality of subtasks required to perform a skill, wherein the plurality of subtasks includes a nondemonstration subtask and at least one demonstration subtask that relies on force feedback and visual characteristics of the workcell, and wherein the demonstration subtask is associated with a pretrained base control policy that was previously trained using at least one different robot located at a different site;

generating actual local demonstration data from a workcell demonstration of the demonstration subtask by operating the physical robot in the workcell while capturing sensor data generated by a plurality of sensors in the workcell of the physical robot, wherein the actual local demonstration data comprises task state representations generated at a base update rate corresponding to a rate at which the physical robot receives commands for execution, and

wherein generating each task state representation comprises processing sensor output of each sensor of the plurality of sensors by a different respective neural network of a plurality of neural networks to generate, by each neural network, a different portion of the task state representation, wherein each portion of the task state representation is generated at a different rate, and wherein generating the task state representation comprises reading, at the base update rate, the most recently updated output of each of the plurality of neural networks processing the sensor data, wherein the actual local demonstration data includes robot state data representing force sensor data obtained during a workcell demonstration of the demonstration subtask;

generating, from the perceptual data, a virtual model of a portion of the workcell;

performing a plurality of simulated training iterations using a virtual robot in the virtual model of the portion of the workcell to generate, by a training system, simulated local demonstration data from the virtual model of the portion of the workcell, including:

simulating respective outputs of the plurality of sensors in the workcell capturing data at different respective rates;

generating, at each time step corresponding to the base update rate, a respective simulated task state representation using the simulated outputs of the plurality of sensors capturing data at different respective rates, including processing the simulated sensor output of each sensor of the plurality of sensors by a different respective neural network of the plurality of neural networks used to generate the actual local demonstration data to generate, by each respective neural network, a different portion of the simulated task state representation, wherein each portion of the simulated task state representation is generated at a different rate, and wherein generating the simulated task state representation comprises obtaining, at the base update rate, the most recently updated output of each of the plurality of neural networks processing the simulated sensor data; and

providing, to a current control policy subsystem at each time step corresponding to the base update rate, a most recently generated simulated task state representation having portions generated by different respective neural networks processing simulated sensor data at different rates to generate one or more commands to be performed by the virtual robot, and

updating the simulated outputs of the plurality of sensors according to the one or more commands generated by the current control policy subsystem; and

tuning the current control policy subsystem for the demonstration subtask using the simulated local demonstration data generated from the virtual model of the portion of the workcell.

2. The method of claim 1 , wherein tuning the base control policy at least partially overlaps with a user collecting the actual local demonstration data from the robot.

3. The method of claim 1 , wherein the perceptual data comprises a camera image, depth camera data, lidar scan data, or laser scan data of the workcell.

4. The method of claim 1 , wherein generating the virtual model of the workcell comprises generating a plurality of different perspectives of a target of the demonstration subtask defined in the skill template.

5. The method of claim 1 , wherein generating the virtual model of the workcell comprises generating a three-dimensional representation of a portion of the workcell represented by the perceptual data.

6. The method of claim 1 , wherein the robot state data of the actual local demonstration data includes joint data of the robot representing joint angles of the robot during the workcell demonstration.

7. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

receiving perceptual data of a workcell at a first site, the workcell having a physical robot to be configured to execute a task according to a skill template, wherein the skill template specifies a plurality of subtasks required to perform a skill, wherein the plurality of subtasks includes a nondemonstration subtask and at least one demonstration subtask that relies on force feedback and visual characteristics of the workcell, and wherein the demonstration subtask is associated with a pretrained base control policy that was previously trained using at least one different robot located at a different site;

generating actual local demonstration data from a workcell demonstration of the demonstration subtask by operating the physical robot in the workcell while capturing sensor data generated by a plurality of sensors in the workcell of the physical robot, wherein the actual local demonstration data comprises task state representations generated at a base update rate corresponding to a rate at which the physical robot receives commands for execution, and

wherein generating each task state representation comprises processing sensor output of each sensor of the plurality of sensors by a different respective neural network of a plurality of neural networks to generate, by each neural network, a different portion of the task state representation, wherein each portion of the task state representation is generated at a different rate, and wherein generating the task state representation comprises reading, at the base update rate, the most recently updated output of each of the plurality of neural networks processing the sensor data, wherein the actual local demonstration data includes robot state data representing force sensor data obtained during a workcell demonstration of the demonstration subtask;

generating, from the perceptual data, a virtual model of a portion of the workcell;

performing a plurality of simulated training iterations using a virtual robot in the virtual model of the portion of the workcell to generate, by a training system, simulated local demonstration data from the virtual model of the portion of the workcell including:

simulating respective outputs of the plurality of sensors in the workcell capturing data at different respective rates;

generating, at each time step corresponding to the base update rate, a respective simulated task state representation using the simulated outputs of the plurality of sensors capturing data at different respective rates, including processing the simulated sensor output of each sensor of the plurality of sensors by a different respective neural network of the plurality of neural networks used to generate the actual local demonstration data to generate, by each respective neural network, a different portion of the simulated task state representation, wherein each portion of the simulated task state representation is generated at a different rate, and wherein generating the simulated task state representation comprises obtaining, at the base update rate, the most recently updated output of each of the plurality of neural networks processing the simulated sensor data; and

providing, to a current control policy subsystem at each time step corresponding to the base update rate, a most recently generated simulated task state representation having portions generated by different respective neural networks processing simulated sensor data at different rates to generate one or more commands to be performed by the virtual robot, and

updating the simulated outputs of the plurality of sensors according to the one or more commands generated by the current control policy subsystem; and

tuning the current control policy subsystem for the demonstration subtask using the simulated local demonstration data generated from the virtual model of the portion of the workcell.

8. The system of claim 7 , wherein tuning the base control policy at least partially overlaps with a user collecting the actual local demonstration data from the robot.

9. The system of claim 7 , wherein the perceptual data comprises a camera image, depth camera data, lidar scan data, or laser scan data of the workcell.

10. The system of claim 7 , wherein generating the virtual model of the workcell comprises generating a plurality of different perspectives of a target of the demonstration subtask defined in the skill template.

11. The system of claim 7 , wherein generating the virtual model of the workcell comprises generating a three-dimensional representation of a portion of the workcell represented by the perceptual data.

12. The system of claim 7 , wherein the robot state data of the actual local demonstration data includes joint data of the robot representing joint angles of the robot during the workcell demonstration.

13. One or more non-transitory computer storage media encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving perceptual data of a workcell at a first site, the workcell having a physical robot to be configured to execute a task according to a skill template, wherein the skill template specifies a plurality of subtasks required to perform a skill, wherein the plurality of subtasks includes a nondemonstration subtask and at least one demonstration subtask that relies on force feedback and visual characteristics of the workcell, and wherein the demonstration subtask is associated with a pretrained base control policy that was previously trained using at least one different robot located at a different site;

generating actual local demonstration data from a workcell demonstration of the demonstration subtask by operating the physical robot in the workcell while capturing sensor data generated by a plurality of sensors in the workcell of the physical robot, wherein the actual local demonstration data comprises task state representations generated at a base update rate corresponding to a rate at which the physical robot receives commands for execution, and

wherein generating each task state representation comprises processing sensor output of each sensor of the plurality of sensors by a different respective neural network of a plurality of neural networks to generate, by each neural network, a different portion of the task state representation, wherein each portion of the task state representation is generated at a different rate, and wherein generating the task state representation comprises reading, at the base update rate, the most recently updated output of each of the plurality of neural networks processing the sensor data, wherein the actual local demonstration data includes robot state data representing force sensor data obtained during a workcell demonstration of the demonstration subtask;

generating, from the perceptual data, a virtual model of a portion of the workcell;

performing a plurality of simulated training iterations using a virtual robot in the virtual model of the portion of the workcell to generate, by a training system, simulated local demonstration data from the virtual model of the portion of the workcell including:

simulating respective outputs of the plurality of sensors in the workcell capturing data at different respective rates,

generating, at each time step corresponding to the base update rate, a respective simulated task state representation using the simulated outputs of the plurality of sensors capturing data at different respective rates, including processing the simulated sensor output of each sensor of the plurality of sensors by a different respective neural network of the plurality of neural networks used to generate the actual local demonstration data to generate, by each respective neural network, a different portion of the simulated task state representation, wherein each portion of the simulated task state representation is generated at a different rate, and wherein generating the simulated task state representation comprises obtaining, at the base update rate, the most recently updated output of each of the plurality of neural networks processing the simulated sensor data; and

providing, to a current control policy subsystem at each time step corresponding to the base update rate, a most recently generated simulated task state representation having portions generated by different respective neural networks processing simulated sensor data at different rates to generate one or more commands to be performed by the virtual robot, and

updating the simulated outputs of the plurality of sensors according to the one or more commands generated by the current policy subsystem; and

tuning the current control policy subsystem for the demonstration subtask using the simulated local demonstration data generated from the virtual model of the portion of the workcell.

14. The one or more non-transitory computer storage media of claim 13 , wherein tuning the base control policy at least partially overlaps with a user collecting actual local demonstration data from the robot.

15. The method of claim 1 , further comprising tuning the current control policy subsystem using the actual local demonstration data generated from the physical robot.

16. The method of claim 1 , wherein the current control policy subsystem is trained using: 1) Data from a different robot at a different site, 2) actual local demonstration data from the physical workcell at the first site, and 3) simulated local demonstration data from the virtual model of the workcell at the first site.

17. The method of claim 16 , further comprising executing the control policy subsystem tuned using the actual local demonstration data and the simulated demonstration data at the first site to generate commands from sensor data that cause the physical robot to perform the demonstration subtask in the physical workcell.