Robust reinforcement learning for continuous control with model misspecification
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a policy neural network having policy parameters. One of the methods includes sampling a mini-batch comprising one or more observation-action-reward tuples generated as a result of interactions of a first agent with a first environment; determining an update to current values of the Q network parameters by minimizing a robust entropy-regularized temporal difference (TD) error that accounts for possible perturbations of the states of the first environment represented by the observations in the observation-action-reward tuples; and determining, using the Q-value neural network, an update to the policy network parameters using the sampled mini-batch of observation-action-reward tuples.
1 . A method of training a policy neural network having a plurality of policy network parameters,
wherein the policy neural network is configured to receive a policy input comprising an observation characterizing a current state of an environment and to process the network input in accordance with the policy network parameters to generate a policy output that defines a probability distribution over a space of possible actions to be performed by an agent interacting with the environment,
wherein the policy neural network is trained jointly with a Q-value neural network (i) having a plurality of Q network parameters and (ii) configured to receive a Q network input comprising data identifying an action and the observation and to process the Q network input in accordance with the Q network parameters to generate a Q value for the action, and
wherein the method comprises:
sampling a mini-batch comprising one or more observation—action—reward tuples generated as a result of interactions of a first agent with a first environment;
determining an update to current values of the Q network parameters comprising:
for each tuple and for each possible perturbation of the environment in a set of a plurality of possible perturbations of the environment:
causing the agent to perform the action in the tuple when the state of the environment represented by the observation in the tuple has been perturbed by applying the possible perturbation to the state of the environment represented by the observation; and
in response, obtaining a next observation characterizing a perturbed next state; and
minimizing a robust entropy-regularized temporal difference (TD) error that accounts for possible perturbations of the states of the first environment represented by the observations in the sampled mini-batch of observation-action-reward tuples and measures, for each tuple, an error between (i) a target Q value determined based on the reward in the tuple and an infimum or average of entropy-regularized Q values for the perturbed next states for the plurality of possible perturbations and (ii) a Q value for the observation-action pair in the tuple; and
determining, using the Q-value neural network, an update to the policy network parameters using the sampled mini-batch of observation-action-reward tuples.
2 . The method of claim 1 , further comprising:
providing data specifying the trained policy neural network for use in controlling a second agent interacting with a second, different environment.
3 . The method of claim 2 , wherein the first environment is a simulation of a real-world environment and the second, different environment is the real-world environment, wherein the first agent is a real-world mechanical agent, and wherein the first agent is a simulation of the second agent.
4 . The method of claim 2 , wherein the first environment is a first real-world environment and the second, different environment is a second, different real-world environment.
5 . The method of claim 4 , wherein the first agent is a first real-world mechanical agent and the second agent is a second, different real-world mechanical agent.
6 . The method of claim 1 , wherein the robust entropy-regularized temporal difference (TD) error measures, for each tuple, an error between (i) a sum of the reward in the tuple and an infimum of entropy-regularized Q values for the perturbed next states and (ii) a Q value for the observation-action pair in the tuple.
7 . The method of claim 1 , wherein the robust entropy-regularized temporal difference (TD) error measures, for each tuple, an error between (i) a sum of the reward in the tuple and an average of entropy-regularized Q values for the perturbed next states and (ii) a Q value for the observation-action pair in the tuple.
8 . The method of claim 6 , further comprising generating a respective entropy-regularized Q value for each of the perturbed next states, comprising:
processing the next observation characterizing the perturbed next state using the policy neural network to generate a next probability distribution over possible actions;
sampling a next action from the next probability distribution;
determining a Q value for the next observation-next action pair;
determining an entropy regularization penalty based on a divergence between the next probability distribution and a reference next probability distribution; and
determining the respective entropy-regularized Q value for the perturbed next state from at least the Q value for the next observation-next action pair and the entropy regularization penalty.
9 . The method of claim 8 , wherein the reference next probability distribution is a probability distribution generated by the policy neural network in accordance with earlier values of the policy parameters.
10 . The method of claim 8 , wherein the Q value for the next observation-next action pair is generated by a target Q neural network having the same architecture as the Q-value neural network but parameter values that change more slowly during training than the Q network parameters.
11 . The method of claim 1 , wherein the possible perturbations include perturbations to one or more of: dimensions of portions of the agent or dimensions of objects in the environment, orientation of portions of the agent, or orientations of objects in the environment.
12 . The method of claim 1 , wherein the policy output comprises output parameters of a probability distribution over a continuous space of actions.
13 . The method of claim 12 , wherein the parameters are means and covariances of a multi-variate Normal distribution over the continuous space of actions.
14 . The method of claim 1 , wherein determining, using the Q network, an update to the policy network parameters using the sampled batch of observation—action—reward tuples comprises applying an actor-critic technique to the sampled batch using the Q network as a critic.
15 . The method of claim 14 , wherein the actor-critic technique is Maximum a Posteriori Policy Optimisation.
16 . The method of claim 14 , wherein the actor-critic technique is stochastic value gradients (SVG).
17 . One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a policy neural network having a plurality of policy network parameters,
wherein the policy neural network is configured to receive a policy input comprising an observation characterizing a current state of an environment and to process the network input in accordance with the policy network parameters to generate a policy output that defines a probability distribution over a space of possible actions to be performed by an agent interacting with the environment,
wherein the policy neural network is trained jointly with a Q-value neural network (i) having a plurality of Q network parameters and (ii) configured to receive a Q network input comprising data identifying an action and the observation and to process the Q network input in accordance with the Q network parameters to generate a Q value for the action, and
wherein the operations comprise:
sampling a mini-batch comprising one or more observation—action—reward tuples generated as a result of interactions of a first agent with a first environment;
determining an update to current values of the Q network parameters, comprising:
for each tuple and for each possible perturbation of the environment in a set of a plurality of possible perturbations of the environment:
causing the agent to perform the action in the tuple when the state of the environment represented by the observation in the tuple has been perturbed by applying the possible perturbation to the state of the environment represented by the observation; and
in response, obtaining a next observation characterizing a perturbed next state; and
minimizing a robust entropy-regularized temporal difference (TD) error that accounts for possible perturbations of the states of the first environment represented by the observations in the sampled mini-batch of observation—action—reward tuples and measures, for each tuple, an error between (i) a target Q value determined based on the reward in the tuple and an infimum or average of entropy-regularized Q values for the perturbed next states for the plurality of possible perturbations and (ii) a Q value for the observation-action pair in the tuple; and
determining, using the Q-value neural network, an update to the policy network parameters using the sampled mini-batch of observation-action-reward tuples.
18 . A system comprising one or more computers and one or more storage devices storing instruction that when executed by one or more computers cause the one or more computers to perform operations fort raining a policy neural network having a plurality of policy network parameters,
wherein the policy neural network is configured to receive a policy input comprising an observation characterizing a current state of an environment and to process the network input in accordance with the policy network parameters to generate a policy output that defines a probability distribution over a space of possible actions to be performed by an agent interacting with the environment,
wherein the policy neural network is trained jointly with a Q-value neural network (i) having a plurality of Q network parameters and (ii) configured to receive a Q network input comprising data identifying an action and the observation and to process the Q network input in accordance with the Q network parameters to generate a Q value for the action, and
wherein the operations comprise:
sampling a mini-batch comprising one or more observation—action—reward tuples generated as a result of interactions of a first agent with a first environment;
determining an update to current values of the Q network parameters, comprising:
for each tuple and for each possible perturbation of the environment in a set of a plurality of possible perturbations of the environment:
causing the agent to perform the action in the tuple when the state of the environment represented by the observation in the tuple has been perturbed by applying the possible perturbation to the state of the environment represented by the observation; and
in response, obtaining a next observation characterizing a perturbed next state; and
minimizing a robust entropy-regularized temporal difference (TD) error that accounts for possible perturbations of the states of the first environment represented by the observations in the sampled mini-batch of observation—action—reward tuples and measures, for each tuple, an error between (i) a target Q value determined based on the reward in the tuple and an infimum or average of entropy-regularized Q values for the perturbed next states for the plurality of possible perturbations and (ii) a Q value for the observation-action pair in the tuple; and
determining, using the Q-value neural network, an update to the policy network parameters using the sampled mini-batch of observation—action—reward tuples.
19 . The system of claim 18 , wherein the robust entropy-regularized temporal difference (TD) error measures, for each tuple, an error between (i) a sum of the reward in the tuple and an infimum of entropy-regularized Q values for the perturbed next states and (ii) a Q value for the observation-action pair in the tuple.
20 . The system of claim 18 , wherein the robust entropy-regularized temporal difference (TD) error measures, for each tuple, an error between (i) a sum of the reward in the tuple and an average of entropy-regularized Q values for the perturbed next states and (ii) a Q value for the observation-action pair in the tuple.