Training action selection neural networks using off-policy actor critic reinforcement learning and stochastic dueling neural networks
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an action selection neural network. One of the methods includes maintaining a replay memory that stores trajectories generated as a result of interaction of an agent with an environment; and training an action selection neural network having policy parameters on the trajectories in the replay memory, wherein training the action selection neural network comprises: sampling a trajectory from the replay memory; and adjusting current values of the policy parameters by training the action selection neural network on the trajectory using an off-policy actor critic reinforcement learning technique.
1. A method comprising:
maintaining a replay memory that stores trajectories generated as a result of interaction of an agent with an environment, each trajectory comprising respective action selection data at each of a plurality of time steps, the action selection data at each time step identifying:
(i) an observation characterizing a state of the environment,
(ii) an action performed by the agent in response to the observation,
(iii) a reward received in response to the agent performing the action, and
(iv) at least an action selection score assigned to the performed action in determining which action to perform in response to the observation; and
training an action selection neural network having policy parameters on the trajectories in the replay memory, wherein the action selection neural network is configured to:
receive a provided observation characterizing a state of the environment; and
process the provided observation to generate a network output that defines a score distribution over possible actions that can be performed by the agent in response to the provided observation, and
wherein training the action selection neural network comprises:
sampling a trajectory from the replay memory; and
adjusting current values of the policy parameters by training the action selection neural network on the trajectory using an off-policy actor critic reinforcement learning technique, comprising:
for each time step in the trajectory:
processing the observation using the action selection neural network to define a main score distribution, wherein the main score distribution defines a distribution over a set of actions that can be performed by the agent to interact with the environment;
sampling a plurality of sampled actions, from the set of actions, in accordance with the main score distribution over the set of actions; and
receiving, by a stochastic dueling neural network, a data input to the stochastic dueling neural network comprising: (i) the observation characterizing the state of the environment at the time step, (ii) the performed action for the time step, and (iii) the plurality of sampled actions sampled in accordance with the main score distribution over the set of actions that can be performed by the agent to interact with the environment;
jointly processing: (i) the observation characterizing the state of the environment at the time step, (ii) the performed action for the time step, and (iii) the plurality of sampled actions sampled in accordance with the main score distribution over the set of actions that can be performed by the agent to interact with the environment, using the stochastic dueling neural network, to generate a stochastic Q value for the performed action, wherein the stochastic Q value for the performed action is a stochastic estimate of a Q value for the performed action; and
determining, based at least in part on the stochastic Q values, an update to the current values of the policy parameters.
2. The method of claim 1 , wherein jointly processing: (i) the observation characterizing the state of the environment at the time step, (ii) the performed action for the time step, and (iii) the plurality of sampled actions sampled in accordance with the main score distribution over the set of actions that can be performed by the agent to interact with the environment to generate the stochastic Q value for the performed action comprises:
generating, using the stochastic dueling neural network, a respective advantage estimate for each of the plurality of sampled actions;
generating, using the stochastic dueling neural network, an advantage estimate for the performed action.
3. The method of claim 2 , wherein jointly processing: (i) the observation characterizing the state of the environment at the time step, (ii) the performed action for the time step, and (iii) the plurality of sampled actions sampled in accordance with the main score distribution over the set of actions that can be performed by the agent to interact with the environment to generate the stochastic Q value for the performed action further comprises:
generating, using the stochastic dueling neural network, a deterministic value estimate for the time step.
4. The method of claim 3 , wherein jointly processing: (i) the observation characterizing the state of the environment at the time step, (ii) the performed action for the time step, and (iii) the plurality of sampled actions sampled in accordance with the main score distribution over the set of actions that can be performed by the agent to interact with the environment to generate the stochastic Q value for the performed action further comprises:
combining: (i) the respective advantage estimate for each of the plurality of sampled actions, (ii) the advantage estimate for the performed action, and (iii) the deterministic value estimate, to generate the stochastic Q value for the performed action.
5. The method of claim 1 , wherein the possible actions that can be performed by the agent in response to the provided observation are represented as points in a continuous action space and the network output comprises distribution parameters that define the score distribution over the points in the continuous action space.
6. The method of claim 1 , wherein determining, based at least in part on the stochastic Q values, an update to the current values of the policy parameters comprises, for each time step in the trajectory from the final time step in the trajectory to the initial time step in the trajectory:
determining a main gradient for the time step based at least in part on an off-policy correction estimate for the time step; and
updating the off-policy correction estimate for the time step based at least in part on the stochastic Q value for the performed action at the time step.
7. The method of claim 6 , wherein determining the main gradient for the time step based at least in part on the off-policy correction estimate for the time step comprises:
determining the main gradient for the time step based at least in part on a product of: (i) a difference between the off-policy correction estimate for the time step and a value estimate for the time step, and (ii) a gradient of a score for the performed action at the time step under the main score distribution for the time step.
8. A system comprising:
one or more computers; and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
maintaining a replay memory that stores trajectories generated as a result of interaction of an agent with an environment, each trajectory comprising respective action selection data at each of a plurality of time steps, the action selection data at each time step identifying:
(i) an observation characterizing a state of the environment,
(ii) an action performed by the agent in response to the observation,
(iii) a reward received in response to the agent performing the action, and
(iv) at least an action selection score assigned to the performed action in determining which action to perform in response to the observation; and
training an action selection neural network having policy parameters on the trajectories in the replay memory, wherein the action selection neural network is configured to:
receive a provided observation characterizing a state of the environment; and
process the provided observation to generate a network output that defines a score distribution over possible actions that can be performed by the agent in response to the provided observation, and
wherein training the action selection neural network comprises:
sampling a trajectory from the replay memory; and
adjusting current values of the policy parameters by training the action selection neural network on the trajectory using an off-policy actor critic reinforcement learning technique, comprising:
for each time step in the trajectory:
processing the observation using the action selection neural network to define a main score distribution, wherein the main score distribution defines a distribution over a set of actions that can be performed by the agent to interact with the environment;
sampling a plurality of sampled actions, from the set of actions, in accordance with the main score distribution over the set of actions; and
receiving, by a stochastic dueling neural network, a data input to the stochastic dueling neural network comprising: (i) the observation characterizing the state of the environment at the time step, (ii) the performed action for the time step, and (iii) the plurality of sampled actions sampled in accordance with the main score distribution over the set of actions that can be performed by the agent to interact with the environment;
jointly processing: (i) the observation characterizing the state of the environment at the time step, (ii) the performed action for the time step, and (iii) the plurality of sampled actions sampled in accordance with the main score distribution over the set of actions that can be performed by the agent to interact with the environment, using the stochastic dueling neural network, to generate a stochastic Q value for the performed action, wherein the stochastic Q value for the performed action is a stochastic estimate of a Q value for the performed action; and
determining, based at least in part on the stochastic Q values, an update to the current values of the policy parameters.
9. The system of claim 8 , jointly processing: (i) the observation characterizing the state of the environment at the time step, (ii) the performed action for the time step, and (iii) the plurality of sampled actions sampled in accordance with the main score distribution over the set of actions that can be performed by the agent to interact with the environment to generate the stochastic Q value for the performed action comprises:
generating, using the stochastic dueling neural network, a respective advantage estimate for each of the plurality of sampled actions;
generating, using the stochastic dueling neural network, an advantage estimate for the performed action.
10. The system of claim 9 , wherein jointly processing: (i) the observation characterizing the state of the environment at the time step, (ii) the performed action for the time step, and (iii) the plurality of sampled actions sampled in accordance with the main score distribution over the set of actions that can be performed by the agent to interact with the environment to generate the stochastic Q value for the performed action further comprises:
generating, using the stochastic dueling neural network, a deterministic value estimate for the time step.
11. The system of claim 10 , wherein jointly processing: (i) the observation characterizing the state of the environment at the time step, (ii) the performed action for the time step, and (iii) the plurality of sampled actions sampled in accordance with the main score distribution over the set of actions that can be performed by the agent to interact with the environment to generate the stochastic Q value for the performed action further comprises:
combining: (i) the respective advantage estimate for each of the plurality of sampled actions, (ii) the advantage estimate for the performed action, and (iii) the deterministic value estimate, to generate the stochastic Q value for the performed action.
12. The system of claim 8 , wherein the possible actions that can be performed by the agent in response to the provided observation are represented as points in a continuous action space and the network output comprises distribution parameters that define the score distribution over the points in the continuous action space.
13. The system of claim 8 , wherein determining, based at least in part on the stochastic Q values, an update to the current values of the policy parameters comprises, for each time step in the trajectory from the final time step in the trajectory to the initial time step in the trajectory:
determining a main gradient for the time step based at least in part on an off-policy correction estimate for the time step; and
updating the off-policy correction estimate for the time step based at least in part on the stochastic Q value for the performed action at the time step.
14. The system of claim 13 , wherein determining the main gradient for the time step based at least in part on the off-policy correction estimate for the time step comprises:
determining the main gradient for the time step based at least in part on a product of: (i) a difference between the off-policy correction estimate for the time step and a value estimate for the time step, and (ii) a gradient of a score for the performed action at the time step under the main score distribution for the time step.
15. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
maintaining a replay memory that stores trajectories generated as a result of interaction of an agent with an environment, each trajectory comprising respective action selection data at each of a plurality of time steps, the action selection data at each time step identifying:
(i) an observation characterizing a state of the environment,
(ii) an action performed by the agent in response to the observation,
(iii) a reward received in response to the agent performing the action, and
(iv) at least an action selection score assigned to the performed action in determining which action to perform in response to the observation; and
training an action selection neural network having policy parameters on the trajectories in the replay memory, wherein the action selection neural network is configured to:
receive a provided observation characterizing a state of the environment; and
process the provided observation to generate a network output that defines a score distribution over possible actions that can be performed by the agent in response to the provided observation, and
wherein training the action selection neural network comprises:
sampling a trajectory from the replay memory; and
adjusting current values of the policy parameters by training the action selection neural network on the trajectory using an off-policy actor critic reinforcement learning technique, comprising:
for each time step in the trajectory:
processing the observation using the action selection neural network to define a main score distribution, wherein the main score distribution defines a distribution over a set of actions that can be performed by the agent to interact with the environment;
sampling a plurality of sampled actions, from the set of actions, in accordance with the main score distribution over the set of actions; and
receiving, by a stochastic dueling neural network, a data input to the stochastic dueling neural network comprising: (i) the observation characterizing the state of the environment at the time step, (ii) the performed action for the time step, and (iii) the plurality of sampled actions sampled in accordance with the main score distribution over the set of actions that can be performed by the agent to interact with the environment;
jointly processing: (i) the observation characterizing the state of the environment at the time step, (ii) the performed action for the time step, and (iii) the plurality of sampled actions sampled in accordance with the main score distribution over the set of actions that can be performed by the agent to interact with the environment, using the stochastic dueling neural network, to generate a stochastic Q value for the performed action, wherein the stochastic Q value for the performed action is a stochastic estimate of a Q value for the performed action; and
determining, based at least in part on the stochastic Q values, an update to the current values of the policy parameters.
16. The non-transitory computer storage media of claim 15 , wherein jointly processing: (i) the observation characterizing the state of the environment at the time step, (ii) the performed action for the time step, and (iii) the plurality of sampled actions sampled in accordance with the main score distribution over the set of actions that can be performed by the agent to interact with the environment to generate the stochastic Q value for the performed action comprises:
generating, using the stochastic dueling neural network, a respective advantage estimate for each of the plurality of sampled actions;
generating, using the stochastic dueling neural network, an advantage estimate for the performed action.
17. The non-transitory computer storage media of claim 16 , wherein jointly processing: (i) the observation characterizing the state of the environment at the time step, (ii) the performed action for the time step, and (iii) the plurality of sampled actions sampled in accordance with the main score distribution over the set of actions that can be performed by the agent to interact with the environment to generate the stochastic Q value for the performed action further comprises:
generating, using the stochastic dueling neural network, a deterministic value estimate for the time step.
18. The non-transitory computer storage media of claim 17 , wherein jointly processing: (i) the observation characterizing the state of the environment at the time step, (ii) the performed action for the time step, and (iii) the plurality of sampled actions sampled in accordance with the main score distribution over the set of actions that can be performed by the agent to interact with the environment to generate the stochastic Q value for the performed action further comprises:
combining: (i) the respective advantage estimate for each of the plurality of sampled actions, (ii) the advantage estimate for the performed action, and (iii) the deterministic value estimate, to generate the stochastic Q value for the performed action.
19. The non-transitory computer storage media of claim 15 , wherein the possible actions that can be performed by the agent in response to the provided observation are represented as points in a continuous action space and the network output comprises distribution parameters that define the score distribution over the points in the continuous action space.
20. The non-transitory computer storage media of claim 15 , wherein determining, based at least in part on the stochastic Q values, an update to the current values of the policy parameters comprises, for each time step in the trajectory from the final time step in the trajectory to the initial time step in the trajectory:
determining a main gradient for the time step based at least in part on an off-policy correction estimate for the time step; and
updating the off-policy correction estimate for the time step based at least in part on the stochastic Q value for the performed action at the time step.