Control policies for robotic agents
Methods, systems, and apparatus, including computer programs encoded on computer storage media, of training a global policy neural network. One of the methods includes initializing a plurality of instances of the robotic task. For each instance of the robotic task, the method includes generating a trajectory of state-action pairs by selecting actions to be performed by the robotic agent while performing the instance of the robotic task in accordance with current values of the parameters of the global policy neural network, and optimizing a local policy controller that is specific to the instance on the trajectory of state-action pairs for the instance. The method further includes generating training data for the global policy neural network using the local policy controllers, and training the global policy neural network on the training data to adjust the current values of the parameters of the global policy neural network.
1. A method of training a global policy neural network used to select actions to be performed by a robotic agent interacting with a real-world environment to perform a robotic task,
wherein the global policy neural network is configured to receive as input an observation characterizing a state of the environment and to generate a global policy output in accordance with current values of a plurality of parameters, and
wherein the method comprises:
initializing a plurality of instances of the robotic task, comprising randomly selecting a respective initial state of the environment for each instance;
for each instance:
generating a trajectory of state-action pairs by selecting actions to be performed by the robotic agent while performing the instance of the robotic task in accordance with current values of the parameters of the global policy neural network, and
optimizing a local policy controller that is specific to the instance on the trajectory of state-action pairs for the instance;
generating training data for the global policy neural network using the local policy controllers; and
training the global policy neural network on the training data to adjust the current values of the parameters of the global policy neural network.
2. The method of claim 1 , wherein optimizing the local policy controller that is specific to the instance on the trajectory of state-action pairs for the instance comprises:
performing an optimization step with path integral stochastic optimal control on the trajectory of state-action pairs for the instance.
3. The method of claim 1 , wherein optimizing the local policy controller that is specific to the instance on the trajectory of state-action pairs for the instance comprises:
performing an optimization step using a linear quadratic regulator on the trajectory of state-action pairs for the instance.
4. The method of claim 1 , wherein, for each instance, the local policy controller that is specific to the instance is a time-varying linear-Gaussian controller.
5. The method of claim 1 , wherein each local policy controller receives low-dimensional data characterizing a state of the environment and generates a local policy output.
6. The method of claim 5 , wherein each observation is high-dimensional data characterizing the state of the environment.
7. The method of claim 6 , wherein each observation includes raw sensor data captured by one or more sensors of the robotic agent.
8. The method of claim 7 , wherein the global policy neural network comprises:
a convolutional sub-neural network configured to receive an image of the environment captured by a camera of the robotic agent and to process the image to generate a plurality of visual features in accordance with a plurality of convolutional parameters; and
a fully-connected sub-neural network configured to receive the visual features and a robot state vector that characterizes a current state of the robotic agent and to process the visual features and the robot state vector to generate the global policy output for the observation in accordance with a plurality of fully-connected parameters.
9. The method of claim 8 , wherein the robot state vector comprises data characterizing a position and a velocity of one or more components of the robotic agent.
10. The method of claim 8 , wherein training the global policy neural network on the training data to adjust the current values of the parameters of the global policy neural network comprises:
adjusting current values of the fully-connected parameters while holding current values of the convolutional parameters fixed.
11. The method of claim 8 , wherein training the global policy neural network on the training data to adjust the current values of the parameters of the global policy neural network comprises:
adjusting current values of the fully-connected parameters and current values of the convolutional parameters.
12. The method of claim 8 , further comprising:
pre-training the convolutional sub-neural network with a proxy pose detection objective to determine pre-trained values of the convolutional parameters.
13. The method of claim 1 , wherein the global policy output defines a distribution over a plurality of possible actions.
14. The method of claim 13 , wherein generating a trajectory of state-action pairs by selecting actions to be performed by the robotic agent while performing the instance of the task in accordance with current values of the parameters of the global policy neural network comprises:
receiving a sequence of observations and, in response to each observation:
processing the observation using the global policy neural network to generate a global policy output for the observation that defines a distribution over the plurality of possible actions in accordance with current values of the parameters; and
sampling an action to be performed by the robotic agent in response to the observation from the distribution defined by the global policy output.
15. The method of claim 1 , wherein generating training data for the global policy neural network using the local policy controllers comprises, for each iteration:
determining a respective target global policy output for each of one or more of the states in the state-action pairs from the trajectory for the iteration using the optimized local policy controller for the iteration.
16. The method of claim 1 , wherein the robotic task has discontinuous dynamics.
17. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a global policy neural network used to select actions to be performed by a robotic agent interacting with a real-world environment to perform a robotic task,
wherein the global policy neural network is configured to receive as input an observation characterizing a state of the environment and to generate a global policy output in accordance with current values of a plurality of parameters, and
wherein the operations comprise:
initializing a plurality of instances of the robotic task, comprising randomly selecting a respective initial state of the environment for each instance;
for each instance:
generating a trajectory of state-action pairs by selecting actions to be performed by the robotic agent while performing the instance of the robotic task in accordance with current values of the parameters of the global policy neural network, and
optimizing a local policy controller that is specific to the instance on the trajectory of state-action pairs for the instance;
generating training data for the global policy neural network using the local policy controllers; and
training the global policy neural network on the training data to adjust the current values of the parameters of the global policy neural network.
18. The system of claim 17 , wherein each local policy controller receives low-dimensional data characterizing a state of the environment and generates a local policy output, and wherein each observation is high-dimensional data characterizing the state of the environment.
19. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a global policy neural network used to select actions to be performed by a robotic agent interacting with a real-world environment to perform a robotic task,
wherein the global policy neural network is configured to receive as input an observation characterizing a state of the environment and to generate a global policy output in accordance with current values of a plurality of parameters, and
wherein the operations comprise:
initializing a plurality of instances of the robotic task, comprising randomly selecting a respective initial state of the environment for each instance;
for each instance:
generating a trajectory of state-action pairs by selecting actions to be performed by the robotic agent while performing the instance of the robotic task in accordance with current values of the parameters of the global policy neural network, and
optimizing a local policy controller that is specific to the instance on the trajectory of state-action pairs for the instance;
generating training data for the global policy neural network using the local policy controllers; and
training the global policy neural network on the training data to adjust the current values of the parameters of the global policy neural network.
20. A method of training a global policy neural network used to select actions to be performed by a robotic agent interacting with a real-world environment to perform a robotic task,
wherein the global policy neural network is configured to receive as input an observation characterizing a state of the environment and to generate a global policy output in accordance with current values of a plurality of parameters,
wherein the global policy output defines a distribution over a plurality of possible actions, and
wherein the method comprises:
initializing a plurality of instances of the robotic task;
for each instance:
generating a trajectory of state-action pairs by selecting actions to be performed by the robotic agent while performing the instance of the robotic task in accordance with current values of the parameters of the global policy neural network, and
optimizing a local policy controller that is specific to the instance on the trajectory of state-action pairs for the instance;
generating training data for the global policy neural network using the local policy controllers; and
training the global policy neural network on the training data to adjust the current values of the parameters of the global policy neural network.