Asynchronous deep reinforcement learning
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for asynchronous deep reinforcement learning. One of the systems includes a plurality of workers, wherein each worker is configured to operate independently of each other worker, and wherein each worker is associated with a respective actor that interacts with a respective replica of the environment during the training of the deep neural network.
1. A system comprising one or more computers, wherein the one or more computers are configured to implement a policy neural network used to control an agent that interacts with an environment by performing actions selected from a predetermined set of actions, wherein:
the policy neural network has a plurality of policy parameters and is configured to process an observation characterizing a state of the environment to generate a respective score for each action in the predetermined set of actions;
the policy neural network has been trained along with a baseline neural network that has a plurality of baseline parameters and that is configured to process an input observation characterizing a state of the environment to generate a baseline score representing an estimated long-term reward received by the agent starting from the state characterized by the input observation; and
the policy neural network and the baseline neural network have been trained using a plurality of workers, wherein:
each worker is configured to operate independently of each other worker;
each worker is associated with a respective actor that interacts with a respective replica of the environment during the training; and
during the training, each worker is configured to repeatedly perform operations comprising:
receiving, from a shared memory accessible by each of the workers, current values of the baseline parameters of the baseline neural network and current values of the policy parameters of the policy neural network;
until the environment replica interacted with by the actor associated with the worker transitions into a state that satisfies particular criteria:
receiving observations characterizing states of the environment replica and, in response to each observation, selecting an action to be performed by the actor associated with the worker in response to the observation in accordance with a respective action selection policy for the worker, wherein the selecting comprises, for each observation:
generating, based on the observation and the current values of the parameters of the policy neural network, a respective score for each action in the predetermined set of actions, and
selecting the current action to be performed by the actor using the respective scores;
for each observation received before the environment replica interacted with by the actor associated with the worker transitions into the state that satisfies the particular criteria:
generating, based on the observation and the current values of the parameters of the baseline network, a corresponding baseline score representing an estimated reward received by the agent starting from the state characterized by the observation;
determining an actual long-term reward corresponding to the observation; and
performing an iteration of a reinforcement learning technique to determine respective current gradients for the baseline and policy networks based on the baseline score and the actual long-term reward corresponding to the observation;
updating respective accumulated gradients for the baseline and policy networks based on the respective current gradients for the baseline and policy networks;
determining whether criteria for updating the current values of the parameters of the policy neural network and the baseline neural network have been satisfied; and
when the criteria have been satisfied:
updating the current values of the parameters of the policy neural network and the baseline neural network, wherein the updating comprises:
determining updated values of the parameters of the policy neural network and the baseline neural network using the updated accumulated gradients, and
storing the updated values of the parameters of the policy neural network and the baseline neural network in the shared memory accessible to each of the plurality of workers.
2. The system of claim 1 , wherein each worker executes independently of each other worker on the same computer.
3. The system of claim 1 , wherein the respective action selection policy for each worker is different from the respective action selection policy for each other worker.
4. The system of claim 1 , wherein the operations performed by each worker further comprise:
when the criteria for updating the current values of the parameters of the policy neural network and the baseline neural network have been satisfied, clearing the updated accumulated gradients.
5. The system of claim 1 , wherein determining whether the criteria for updating the current values of the parameters of the policy neural network and the baseline neural network have been satisfied comprises determining that a specified number of iterations of the reinforcement learning technique have been performed since a preceding update of the values of the parameters of the policy neural network and the baseline neural network.
6. The system of claim 1 , wherein determining the actual long-term reward corresponding to the observation comprises:
identifying an actual reward resulting from the actor performing the selected action in response to the observation; and
determining the actual long-term reward corresponding to the observation from at least the actual reward for the observation and a baseline score for the observation characterizing the state that satisfies the particular criteria.
7. The system of claim 6 , wherein determining the actual long-term reward corresponding to the observation from at least the actual reward for the observation and a baseline score for the observation characterizing the state that satisfies the particular criteria comprises:
for the last observation received before the criteria are satisfied, setting the actual long-term reward based on the actual reward for the observation and the baseline score for the observation characterizing the state that satisfies the particular criteria; and
for each other observation received before the criteria are satisfied, setting the actual long-term reward for the observation as the actual reward plus a discounted sum of the actual long-term rewards for observations received after the observation.
8. The system of claim 1 , wherein the current gradients for the policy network are determined as:
∇ θ′ log π( a t |s t ;θ′)( R t −b t )
wherein π(a t |s t ;θ′) represents the score for the action a t performed in response to the observation s t determined in accordance with the current values of the parameters of the policy neural network θ′, R t is the actual long-term reward corresponding to the observation, and b t is the baseline score corresponding to the observation.
9. The system of claim 1 , wherein the current gradients for the baseline neural network are determined as:
∂
(
R
t
-
b
t
)
2
∂
θ
b
′
wherein R t is the actual long-term reward corresponding to the observation b t is the baseline score corresponding to the observation, and θ′ b are the current values of the baseline parameters.
10. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to implement a policy neural network used to control an agent that interacts with an environment by performing actions selected from a predetermined set of actions, wherein:
the policy neural network has a plurality of policy parameters and is configured to process an observation characterizing a state of the environment to generate a respective score for each action in the predetermined set of actions;
the policy neural network has been trained along with a baseline neural network that has a plurality of baseline parameters and that is configured to process an input observation characterizing a state of the environment to generate a baseline score representing an estimated long-term reward received by the agent starting from the state characterized by the input observation; and
the policy neural network and the baseline neural network have been trained using a plurality of workers, wherein:
each worker is configured to operate independently of each other worker;
each worker is associated with a respective actor that interacts with a respective replica of the environment during the training; and
during the training, each worker is configured to repeatedly perform operations comprising:
receiving, from a shared memory accessible by each of the workers, current values of the baseline parameters of the baseline neural network and current values of the policy parameters of the policy neural network;
until the environment replica interacted with by the actor associated with the worker transitions into a state that satisfies particular criteria:
receiving observations characterizing states of the environment replica and, in response to each observation, selecting an action to be performed by the actor associated with the worker in response to the observation in accordance with a respective action selection policy for the worker, wherein the selecting comprises, for each observation:
generating, based on the observation and the current values of the parameters of the policy neural network, a respective score for each action in the predetermined set of actions, and
selecting the current action to be performed by the actor using the respective scores;
for each observation received before the environment replica interacted with by the actor associated with the worker transitions into the state that satisfies the particular criteria:
generating, based on the observation and the current values of the parameters of the baseline network, a corresponding baseline score representing an estimated reward received by the agent starting from the state characterized by the observation;
determining an actual long-term reward corresponding to the observation; and
performing an iteration of a reinforcement learning technique to determine respective current gradients for the baseline and policy networks based on the baseline score and the actual long-term reward corresponding to the observation;
updating respective accumulated gradients for the baseline and policy networks based on the respective current gradients for the baseline and policy networks;
determining whether criteria for updating the current values of the parameters of the policy neural network and the baseline neural network have been satisfied; and
when the criteria have been satisfied:
updating the current values of the parameters of the policy neural network and the baseline neural network, wherein the updating comprises:
determining updated values of the parameters of the policy neural network and the baseline neural network using the updated accumulated gradients, and
storing the updated values of the parameters of the policy neural network and the baseline neural network in the shared memory accessible to each of the plurality of workers.
11. The non-transitory computer storage media of claim 10 , wherein each worker executes independently of each other worker on the same computer.
12. The non-transitory computer storage media of claim 10 , wherein the respective action selection policy for each worker is different from the respective action selection policy for each other worker.
13. The non-transitory computer storage media of claim 10 , wherein the operations performed by each worker further comprise:
when the criteria for updating the current values of the parameters of the policy neural network and the baseline neural network have been satisfied, clearing the updated accumulated gradients.
14. The non-transitory computer storage media of claim 10 , wherein determining whether the criteria for updating the current values of the parameters of the policy neural network and the baseline neural network have been satisfied comprises determining that a specified number of iterations of the reinforcement learning technique have been performed since a preceding update of the values of the parameters of the policy neural network and the baseline neural network.
15. A method performed by one or more data processing apparatus for controlling an agent that interacts with an environment by performing actions selected from a predetermined set of actions, the method comprising:
processing an observation characterizing a state of the environment using a policy neural network having a plurality of policy parameters to generate a respective score for each action in the predetermined set of actions; and
selecting an action to be performed by the agent using the respective score for each action in the predetermined set of actions;
wherein the policy neural network has been trained along with a baseline neural network that has a plurality of baseline parameters and that is configured to process an input observation characterizing a state of the environment to generate a baseline score representing an estimated long-term reward received by the agent starting from the state characterized by the input observation;
wherein the policy neural network and the baseline neural network have been trained using a plurality of workers, wherein:
each worker is configured to operate independently of each other worker;
each worker is associated with a respective actor that interacts with a respective replica of the environment during the training; and
during the training, each worker is configured to repeatedly perform operations comprising:
receiving, from a shared memory accessible by each of the workers, current values of the baseline parameters of the baseline neural network and current values of the policy parameters of the policy neural network;
until the environment replica interacted with by the actor associated with the worker transitions into a state that satisfies particular criteria:
receiving observations characterizing states of the environment replica and, in response to each observation, selecting an action to be performed by the actor associated with the worker in response to the observation in accordance with a respective action selection policy for the worker, wherein the selecting comprises, for each observation:
generating, based on the observation and the current values of the parameters of the policy neural network, a respective score for each action in the predetermined set of actions, and
selecting the current action to be performed by the actor using the respective scores;
for each observation received before the environment replica interacted with by the actor associated with the worker transitions into the state that satisfies the particular criteria:
generating, based on the observation and the current values of the parameters of the baseline network, a corresponding baseline score representing an estimated reward received by the agent starting from the state characterized by the observation;
determining an actual long-term reward corresponding to the observation; and
performing an iteration of a reinforcement learning technique to determine respective current gradients for the baseline and policy networks based on the baseline score and the actual long-term reward corresponding to the observation;
updating respective accumulated gradients for the baseline and policy networks based on the respective current gradients for the baseline and policy networks;
determining whether criteria for updating the current values of the parameters of the policy neural network and the baseline neural network have been satisfied; and
when the criteria have been satisfied:
updating the current values of the parameters of the policy neural network and the baseline neural network, wherein the updating comprises:
determining updated values of the parameters of the policy neural network and the baseline neural network using the updated accumulated gradients, and
storing the updated values of the parameters of the policy neural network and the baseline neural network in the shared memory accessible to each of the plurality of workers.
16. The method of claim 15 , wherein each worker executes independently of each other worker on the same computer.
17. The method of claim 15 , wherein the respective action selection policy for each worker is different from the respective action selection policy for each other worker.
18. The method of claim 15 , wherein the operations performed by each worker further comprise:
when the criteria for updating the current values of the parameters of the policy neural network and the baseline neural network have been satisfied, clearing the updated accumulated gradients.
19. The method of claim 15 , wherein determining whether the criteria for updating the current values of the parameters of the policy neural network and the baseline neural network have been satisfied comprises determining that a specified number of iterations of the reinforcement learning technique have been performed since a preceding update of the values of the parameters of the policy neural network and the baseline neural network.
20. The method of claim 15 , wherein determining the actual long-term reward corresponding to the observation comprises:
identifying an actual reward resulting from the actor performing the selected action in response to the observation; and
determining the actual long-term reward corresponding to the observation from at least the actual reward for the observation and a baseline score for the observation characterizing the state that satisfies the particular criteria.