Reinforcement learning using advantage estimates
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for computing Q values for actions to be performed by an agent interacting with an environment from a continuous action space of actions. In one aspect, a system includes a value subnetwork configured to receive an observation characterizing a current state of the environment and process the observation to generate a value estimate; a policy subnetwork configured to receive the observation and process the observation to generate an ideal point in the continuous action space; and a subsystem configured to receive a particular point in the continuous action space representing a particular action; generate an advantage estimate for the particular action; and generate a Q value for the particular action that is an estimate of an expected return resulting from the agent performing the particular action when the environment is in the current state.
1. A method of training a policy neural network of a reinforcement learning system that is configured to select actions to be performed by an agent interacting with an environment from a set of actions that lie on a continuous domain, the method being performed on a computing system configured to receive training observations representative of the environment, the method comprising:
obtaining an experience tuple identifying i) a training observation characterizing a training state of the environment, ii) a training action performed by the agent in response to the training observation, iii) a reward received as a result of the agent performing the training action in response to the training observation, and iv) a subsequent observation characterizing a subsequent state of the environment; and
training the policy neural network on training data comprising the experience tuple, the training comprising:
processing the training observation using a value neural network and in accordance with current values of parameters of the value neural network to generate a first value estimate, the first value estimate being an estimate of an expected return resulting from the environment being in the training state characterized by the training observation irrespective of which action is performed in response to the training observation;
providing the training observation as input to the policy neural network;
for the training observation, obtaining, as output from the policy neural network and generated in accordance with current values of the parameters of the policy neural network, an output action in the set of actions that lie on the continuous domain;
determining a distance in the continuous domain between i) the output action in the set of all possible actions that lie on the continuous domain that is obtained as output from the policy neural network by processing the training observation and ii) the training action that was performed by the agent in response to the training observation;
generating an advantage estimate for the training action that was performed by the agent in response to the training observation from the determined distance in the continuous domain between i) the output action in the set of actions that lie on the continuous domain that is obtained as output from the policy neural network by processing the training observation and ii) the training action that was performed by the agent in response to the training observation, comprising,
processing, by a function parameter neural network of the reinforcement leaning system, the training observation to generate an output that defines values of a set of state-dependent parameters; and
applying a function having the set of state-dependent parameters to the distance between the output action in the set of actions that lie on the continuous domain and the training action that was performed in response to the training observation, wherein the advantage estimate satisfies:
A
(
x
,
u
θ
A
)
=
-
1
2
(
u
-
μ
(
x
θ
μ
)
)
T
P
(
x
θ
P
)
(
u
-
μ
(
x
θ
μ
)
)
where (u−μ(x|θ μ )) T is a transpose of the distance between the output action in the set of actions that lie on the continuous domain and the training action that was performed in response to the training observation P is a state-dependent parameter matrix that has entries defined by the values of the set of state-dependent parameters, and (u−μ(x|θ μ )) the distance between the output action in the set actions that lie on the continuous domain and the training action that was performed in response to the training observation; and
generating a Q value for the training action performed in response to the training observation by combining the advantage estimate for the training action performed in response to the training observation and the first value estimate that is an estimate of an expected return resulting from the environment being in the training state characterized by the training observation irrespective of which action is performed in response to the training observation;
processing the subsequent observation using the value neural network to generate a new value estimate for the subsequent state, the new value estimate being an estimate of an expected return resulting from the environment being in the subsequent state;
combining the reward and the new value estimate to generate a target Q value for the training action;
determining an update to the current values of the parameters of the policy neural network and the current values of the parameters of the value neural network using an error between i) the Q value for the training action generated using the distance between the output action and the training action and ii) the target Q value; and
after the training, providing the trained policy neural network for use in controlling a mechanical agent interacting with a real-world environment.
2. The method of claim 1 , wherein the agent is the mechanical agent and the environment is the real-world environment.
3. The method of claim 1 , wherein P is a positive definite matrix, and wherein P(x|θ P )=LL T , where L is a lower-triangular matrix having entries derived from the output of the function parameter neural network.
4. The method of claim 1 , further comprising:
generating imagination rollouts, wherein each imagination rollout is a synthetic experience tuple; and
adding the generated imagination rollouts to a replay memory of the reinforcement learning system storing experience tuples used to train the policy neural network and the value neural network.
5. The method of claim 4 , wherein generating an imagination rollout comprises:
identifying a particular observation characterizing a state transitioned into by the environment in response to an action performed by the agent;
selecting an action different from the action performed by the agent in response to the particular observation;
processing the particular observation and the selected action using a state transition model to determine a next observation characterizing a next state that the environment would have transitioned into if the agent had performed the selected action in response to the particular observation; and
generating a synthetic experience tuple that includes the particular observation, the selected action, and the next observation.
6. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, the system configured to cause the one or more computers to perform a method for training a policy neural network of a reinforcement learning system, wherein the policy neural network is configured to select actions to be performed by an agent interacting with an environment from a set of actions that lie on a continuous domain, the method comprising:
obtaining an experience tuple identifying i) a training observation characterizing a training state of the environment, ii) a training action performed by the agent in response to the training observation, iii) a reward received as a result of the agent performing the training action in response to the training observation, and iv) a subsequent observation characterizing a subsequent state of the environment;
training the value neural network on training data comprising the experience tuple, the training comprising:
processing the training observation using a value neural network and in accordance with current values of parameters of the value neural network to generate a first value estimate, the first value estimate being an estimate of an expected return resulting from the environment being in the training state characterized by the training observation irrespective of which action is performed in response to the training observations;
providing the training observation as input to the policy neural network;
for the training observation, obtaining, as output from the policy neural network and generated in accordance with current values of the parameters of the policy neural network, an output action in the set of actions that lie on the continuous domain;
determining a distance in the continuous domain between i) the output action in the set of actions that lie on the continuous domain that is obtained as output from the policy neural network by processing the training observation and ii) the training action that was performed by the agent in response to the training observation;
generating an advantage estimate for the training action that was performed by the agent in response to the training observation from the determined distance in the continuous domain between i) the output action in the set of actions that lie on the continuous domain that is obtained as output from the policy neural network by processing the training observation and ii) the training action that was performed by the agent in response to the training observation, comprising:
processing, by a function parameter neural network of the reinforcement learning system, the training observation to generate an output that defines values, of a set of state-dependent parameters; and
applying a function having the set of state-dependent parameters to the distance between the output action in the set of actions that lie on the continuous domain and the training action that was performed in response to the training observation, wherein the advantage estimate satisfies:
A
(
x
,
u
θ
A
)
=
-
1
2
(
u
-
μ
(
x
θ
μ
)
)
T
P
(
x
θ
P
)
(
u
-
μ
(
x
θ
μ
)
)
where (u−μ(x|θ μ )) T is a transpose of the distance between the output action in the set of actions that lie on the continuous domain and the training action that was performed in response to the training observation, P is a state-dependent parameter matrix that has entries defined by the values of the set of state-dependent parameters, and (u−μ(x|θ μ )) is the distance between the output action in the set of actions that lie on the continuous domain and the training action that was performed in response to the training observation; and
generating a Q value for the training action performed in response to the training observation by combining the advantage estimate for the training action performed in response to the training observation and the first value estimate that is an estimate of an expected return resulting from the environment being in the training state characterized by the training observation irrespective of which action is performed in response to the training observation;
processing the subsequent observation using the value neural network to generate a new value estimate for the subsequent state, the new value estimate being an estimate of an expected return resulting from the environment being in the subsequent state;
combining the reward and the new value estimate to generate a target Q value for the training action;
determining an update to the current values of the parameters of the policy neural network and the current values of the parameters of the value neural network using an error between i) the Q value for the training action generated using the distance between the output action and the training action and ii) the target Q value; and
after the training, providing the trained policy neural network for use in controlling a mechanical agent interacting with a real-world environment.
7. The system of claim 6 , wherein P is a positive definite matrix, and wherein P(x|θ P ))=LL T , where L is a lower-triangular matrix having entries derived from the output of the function parameter neural network.
8. The system of claim 6 , wherein the operations further comprise:
generating imagination rollouts, wherein each imagination rollout is a synthetic experience tuple; and
adding the generated imagination rollouts to a replay memory of the reinforcement learning system storing experience tuples used to train the policy neural network and the value neural network.
9. One or more non-transitory computer storage media encoded with computer program instructions configured to be executed by a plurality of computers that when executed, perform a method for training a policy neural network of a reinforcement learning system, wherein the policy neural network is configured to select actions to be performed by an agent interacting with an environment from a set of actions that lie on a continuous domain, the plurality of computers configured to execute the method comprising steps of:
obtaining an experience tuple identifying i) a training observation characterizing a training state of the environment, ii) a training action performed by the agent in response to the training observation, iii) a reward received as a result of the agent performing the training action in response to the training observation, and iv) a subsequent observation characterizing a subsequent state of the environment;
training the policy neural network on training data comprising the experience tuple, the training comprising:
processing the training observation using a value neural network and in accordance with current values of parameters of the value neural network to generate a first value estimate, the first value estimate being an estimate of an expected return resulting from the environment being in the training state characterized by the training observation irrespective of which action is performed in response to the training observation;
providing the training observation as input to the policy neural network;
for the training observation, obtaining, as output from the policy neural network and generated in accordance with current values of the parameters of the policy neural network, an output action in the set of actions that lie on the continuous domain;
determining a distance in the continuous domain between i) the output action in the set of that lie on the continuous domain that is obtained as output from the policy neural network by processing the training observation and ii) the training action that was performed by the agent in response to the training observation;
generating an advantage estimate for the training action that was performed by the agent in response to the training observation from the determined distance in the continuous domain between i) the output action in the set of actions that lie on the continuous domain that is obtained as output from the policy neural network by processing the training observation and ii) the training action that was performed by the agent in response to the training observation, comprising:
processing, by a function parameter neural network of the reinforcement learning system, the training observation to generate an output that defines values of a set of state-dependent parameters; and
applying a function having the set of state-dependent parameters to the distance between the output action in the set of actions that lie on the continuous domain and the training action that was performed in response to the training observation, wherein the advantage estimate satisfies,
A
(
x
,
u
θ
A
)
=
-
1
2
(
u
-
μ
(
x
θ
μ
)
)
T
P
(
x
θ
P
)
(
u
-
μ
(
x
θ
μ
)
)
where (u−μ(x|θ μ )) T is a transpose of the distance between the output action in the set of actions that lie on the continuous domain and the training action that was performed in response to the training observation, P is a state-dependent parameter matrix that has entries defined by the values of the set of state-dependent parameters, and (u−μ(x|θ μ )) the distance between the output action in the set of actions that lie on the continuous domain and the training action that was performed in response to the training, observation; and
generating a Q value for the training action performed in response to the training observation by combining the advantage estimate for the training action performed in response to the training observation and the first value estimate that is an estimate of an expected return resulting from the environment being in the training state characterized by the training observation irrespective of which action is performed in response to the training observation;
processing the subsequent observation using the value neural network to generate a new value estimate for the subsequent state, the new value estimate being an estimate of an expected return resulting from the environment being in the subsequent state;
combining the reward and the new value estimate to generate a target Q value for the training action;
determining an update to the current values of the parameters of the policy neural network and the current values of the parameters of the value neural network using an error between i) the Q value for the training action generated using the distance between the output and the training action and ii) the target Q value; and
after the training, providing the trained policy neural network for use in controlling a mechanical agent interacting with a real-world environment.
10. The non-transitory computer storage media of claim 9 , wherein P is a positive definite matrix, and wherein P(x|θ P ))=LL T , where L is a lower-triangular matrix having entries derived from the output of the function parameter neural network.
11. The method of claim 1 , wherein the advantage estimate for the training action that was performed by the agent in response to the training observation is an estimate of the expected return of performing the training action in the training state, relative to the first value estimate.
12. The method of claim 1 , wherein upon receiving a new observation, the method further comprising:
processing the new observation using the policy neural network to generate a new output action that lies on the continuous domain;
selecting a new action to be performed by the agent using the output action;
receiving a new reward and a new observation that characterizes a new subsequent state of the environment that the environment transitioned into as a result of the agent performing the new action; and
generating a new experience tuple that includes the new observation, the new action, the new reward, and the new subsequent observation.
13. The system of claim 8 , wherein generating an imagination rollout comprises:
identifying a particular observation characterizing a state transitioned into by the environment in response to an action performed by the agent;
selecting an action different from the action performed by the agent in response to the particular observation;
processing the particular observation and the selected action using a state transition model to determine a next observation characterizing a next state that the environment would have transitioned into if the agent had performed the selected action in response to the particular observation; and
generating a synthetic experience tuple that includes the particular observation, the selected action, and the next observation.
14. The system of claim 6 , wherein upon receiving a new observation, the operations further comprise:
processing the new observation using the policy neural network to generate a new output action that lies on the continuous domain;
selecting a new action to be performed by the agent using the output action;
receiving a new reward and a new observation that characterizes a new subsequent state of the environment that the environment transitioned into as a result of the agent performing the new action; and
generating a new experience tuple that includes the new observation, the new action, the new reward, and the new subsequent observation.
15. The system of claim 6 , wherein the agent is the mechanical agent and the environment is the real-world environment.