Data-efficient reinforcement learning for continuous control tasks

1. A system for training an actor neural network used to select actions to be performed by an agent that interacts with an environment by receiving observations characterizing states of the environment and, in response to each observation, performing an action selected from a continuous space of possible actions, wherein the actor neural network maps observations to next actions in accordance with values of parameters of the actor neural network, and wherein the system comprises:

a plurality of workers, each of the plurality of workers having access to a shared memory configured to store current parameters of the actor neural network,

wherein each worker is configured to operate independently of each other worker,

wherein each worker is configured to communicate with and provide instructions to a respective agent replica that interacts with a respective replica of the environment during the training of the actor neural network,

wherein each worker is configured to repeatedly perform (i) a set of updating operations comprising determining that a threshold number of writes to the shared memory have occurred since a preceding update to the values of the parameters of the actor neural network, and in response to the determination, updating current values of the parameters of the actor neural network in the shared memory, and (ii) a set of acting operations to control the respective agent replica using the actor neural network to perform an action in the respective replica of the environment in order to generate training data for training the actor neural network,

wherein the set of the acting operations comprises:

receiving a current observation characterizing a current state of the environment replica interacted with by the agent replica associated with the worker,

selecting a current action to be performed by the agent replica associated with the worker in response to the current observation using the actor neural network and in accordance with the current values of the parameters,

identifying an actual reward resulting from the agent replica performing the current action in response to the current observation, and

receiving a next observation characterizing a next state of the environment replica interacted with by the agent replica, wherein the environment replica transitioned into the next state from the current state in response to the agent replica performing the current action, and

wherein each worker performs a plurality of iterations of the set of updating operations after selecting the current action and prior to selecting a new action in response to the next observation.

2. The system of claim 1 , the set of updating operations comprising:

determining current values of the parameters of the actor neural network and of a critic neural network from the shared memory accessible by each of the plurality of workers;

obtaining a minibatch of experience tuples from a replay memory, each experience tuple comprising a training observation characterizing a training state of the environment, a training action from the continuous space of actions performed by one of the agent replicas in response to the training observation, a training reward received by the agent replica for performing the training action, and a next training observation characterizing a next training state of the environment;

determining updates to the current values of the parameters of the actor neural network and to the current values of the parameters of the critic neural network using the minibatch of experience tuples; and

writing the updates to the current values of the parameters of the actor neural network and to the current values of the parameters of the critic neural network to the shared memory.

3. The system of claim 2 , the acting operations further comprising:

generating a new experience tuple that includes the current observation, the current action, the actual reward, and the next observation, and

storing the new experience tuple in the replay memory.

4. The system of claim 2 , wherein selecting the new action to be performed in response to the next observation comprises:

determining new values of the parameters of the actor neural network from the shared memory;

processing the new observation using the actor neural network in accordance with the new values of the parameters to map the new observation to a new next action; and

selecting the new action to be performed based on the new next action.

5. The system of claim 1 , wherein each worker executes independently of each other worker on the same computer.

6. The system of claim 1 , wherein each worker executes independently of each other worker on different computers.

7. The system of claim 3 , the acting operations further comprising:

determining that the next state that the environment transitioned to is a state in which a subtask of a task being performed by the agent replica has been completed; and

assigning the actual reward to be a reward associated with completion of the subtask by a composite reward function.

8. The system of claim 1 , wherein the agent replica interacts with the environment replica to complete a task, and wherein the acting operations further comprise:

identifying a trajectory of actions and corresponding states that resulted in successful completion of the task by the agent replica or by another actor; and

initializing an initial state of the agent replica and the environment to a state that matches one of the states in the trajectory.

9. The system of claim 8 , wherein initializing the initial state of the agent replica and the environment to a state that matches one of the states in the trajectory comprises:

selecting one of the states in the trajectory randomly; and

initializing an initial state of the agent replica and the environment to a state that matches the selected state.

10. The system of claim 1 , wherein each worker operates asynchronously from each other worker and wherein each worker writes to and reads from a shared memory storing the current values of the parameters asynchronously from each other worker.

11. The system of claim 1 , wherein the plurality of iterations is a predetermined number of iterations greater than one.