Neural networks for selecting actions to be performed by a robotic agent

1. A system comprising:

a neural network system implemented by one or more computers,

wherein the neural network system is configured to receive an observation characterizing a current state of a real-world environment being interacted with by a robotic agent to perform a robotic task and to process the observation to generate a policy output that defines an action to be performed by the robotic agent in response to the observation,

wherein the neural network system comprises a sequence of deep neural networks (DNNs), and

wherein the sequence of DNNs comprises:

a simulation-trained DNN that has been trained on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment to perform a simulated version of the robotic task, wherein:

the simulation-trained DNN comprises a first plurality of indexed layers, and

the simulation-trained DNN is configured to receive the observation and process the observation through each layer in the first plurality of indexed layers to generate a respective layer output for each layer in the first plurality of indexed layers; and

a first robot-trained DNN that has been trained on interactions of the robotic agent with the real-world environment to perform the robotic task to determine trained values of parameters of the first robot-trained DNN while holding trained values of the parameters of the simulation-trained DNN fixed, wherein

the first robot-trained DNN comprises a second plurality of indexed layers,

the first robot-trained DNN is configured to receive the observation and to process the observation through each layer in the second plurality of indexed layers to generate the policy output, and

one or more of the layers in the second plurality of indexed layers are each configured to receive as input (i) a layer output generated by a preceding layer of the first robot-trained DNN, and (ii) a layer output generated by a preceding layer of the simulation- trained DNN, wherein a preceding layer is a layer whose index is one less than the index of the layer.

2. The system of claim 1 , wherein the first robot-trained DNN has been trained on interactions of the robotic agent with the real-world environment to perform the robotic task.

3. The system of claim 1 , wherein the first robot-trained DNN has fewer parameters than the simulation-trained DNN.

4. The system of claim 1 , wherein, for each of one or more layers of the second plurality of indexed layers, a corresponding layer in the first plurality of indexed layers having a same index is a neural network layer of a same type but of a larger dimension than the layer in the second plurality of indexed layers.

5. The system of claim 1 , wherein the first plurality of indexed layers and the second plurality of indexed layers each include a respective recurrent neural network layer.

6. The system of claim 5 , wherein the recurrent neural network layer in the second plurality of indexed layers is configured to receive as input (i) a layer output of a layer preceding the recurrent neural network layer in the first plurality of indexed layers, (ii) an internal state of the recurrent neural network layer in the first plurality of indexed layers, and (iii) a layer output of a layer preceding the recurrent neural network layer in the second plurality of indexed layers.

7. The system of claim 1 , wherein each of the one or more of the layers in the second plurality of indexed layers that are configured to receive as input (i) a layer output generated by a preceding layer of the first robot-trained DNN, and (ii) a layer output generated by a preceding layer of the simulation-trained DNN is further configured to:

apply a respective first set of parameters to the layer output generated by the preceding layer of the first robot-trained DNN; and

apply a respective second set of parameters to the layer output generated by the preceding layer of the simulation-trained DNN.

8. The system of claim 1 , wherein the policy output defines a respective change in position of each of a plurality of degrees of freedom of the robotic agent.

9. The system of claim 1 , wherein the plurality of degrees of freedom include one or more joints of the robotic agent and one or more actuators of the robotic agent.

10. The system of claim 1 , further comprising:

an action subsystem configured to:

select an action to be performed by the robotic agent based on the policy output; and

cause the robotic agent to perform the selected action.

11. The system of claim 1 , wherein the sequence of DNNs further comprises:

a second robot-trained DNN, wherein:

the second robot-trained DNN comprises a third plurality of indexed layers, and

one or more of the layers in the third plurality of indexed layers are each configured to receive as input (i) a layer output generated by a preceding layer of the first robot-trained DNN, (ii) a layer output generated by a preceding layer of the simulation-trained DNN, and (iii) a layer output generated by a preceding layer of the second robot-trained DNN.

12. The system of claim 11 , wherein:

the second robot-trained DNN is configured to receive different data characterizing the current state in conjunction with the observation; and

the second robot-trained DNN is configured to process the different data through the third plurality of indexed layers to generate a second policy output that defines an action to be performed by the robotic agent to perform a second, different robotic task.

13. The system of claim 12 , wherein the observation is visual data and the different data is proprioceptive data.

14. A method of training a neural network system comprising a sequence of deep neural networks (DNNs) to determine trained values of parameters of one or more of the DNNs in the sequence of DNNs, wherein the neural network system is configured to receive an observation characterizing a current state of a real-world environment being interacted with by a robotic agent to perform a robotic task and to process the observation to generate a policy output that defines an action to be performed by the robotic agent in response to the observation,

wherein the sequence of DNNs comprises:

a simulation-trained DNN that has been trained on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment to perform a simulated version of the robotic task, wherein the simulation-trained DNN comprises a first plurality of indexed layers and is configured to receive the observation and process the observation through each layer in the first plurality of indexed layers to generate a respective layer output for each layer in the first plurality of indexed layers; and

a first robot-trained DNN, wherein

the first robot-trained DNN comprises a second plurality of indexed layers,

the first robot-trained DNN is configured to receive the observation and to process the observation through each layer in the second plurality of indexed layers to generate the policy output, and

one or more of the layers in the second plurality of indexed layers are each configured to receive as input (i) a layer output generated by a preceding layer of the first robot-trained DNN, and (ii) a layer output generated by a preceding layer of the simulation- trained DNN, wherein a preceding layer is a layer whose index is one less than the index of the layer,

wherein the method comprises:

training the first robot-trained DNN on interactions of the robotic agent with the real-world environment to perform the robotic task to determine trained values of parameters of the first robot-trained DNN while holding trained values of the parameters of the simulation-trained DNN fixed.

15. The method of claim 14 ,

wherein an output layer of the first robot-trained DNN is configured to:

receive as input (i) a layer output generated by a layer preceding the output layer of the first robot-trained DNN, and (ii) a layer output generated by a layer preceding an output layer of the simulation-trained DNN;

apply a first set of parameters to the layer output generated by a layer preceding the output layer the first robot-trained DNN; and

apply a second set of parameters to the layer output generated by a layer preceding the output layer of the simulation-trained DNN, and

wherein the method further comprises:

initializing values of the second set of parameters to match trained values of parameters of the output layer of the simulation-trained DNN.

16. The method of claim 15 , further comprising:

initializing values of the first set of parameters to zero.

17. The method of claim 15 , further comprising:

initializing values of parameters of layers of the first robot-trained DNN other than the output layer to random values.

18. The method of claim 14 , further comprising:

training the simulation-trained DNN on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment to perform a simulated version of the robotic task to determine trained values of parameters of the simulation-trained DNN.

19. The method of claim 18 , wherein training the simulation-trained DNN comprises training the simulation-trained DNN on the interactions of the simulated version of the robotic agent using a first reinforcement learning technique.

20. The method of claim 19 , wherein training the first robot-trained DNN comprises training the first robot-trained DNN on the interactions of the robotic agent using a second reinforcement learning technique.