Controlling agents over long time scales using temporal value transport

1. A method of training a neural network system augmented with an external memory and used to control an agent interacting with an environment to perform a specified task, the method comprising:

causing the agent to perform a task episode in which the agent attempts to perform the specified task, comprising, at each of a sequence of time steps during the task episode:

obtaining an observation characterizing a current state of the environment;

for each of one or more read heads, reading data from the external memory;

writing data to the external memory;

selecting, from the observation and using the neural network system, an action to be performed by the agent in response to the observation, comprising selecting the action using at least (i) data read from the external memory at a preceding time step or (ii) data read from the external memory at the time step;

generating, using the neural network system, a value prediction representing a value of the environment being in the current state to successfully performing the specified task, comprising generating the value prediction using at least (i) data read from the external memory at a preceding time step or (ii) data read from the external memory at the time step;

causing the agent to perform the selected action; and

in response, receiving an actual reward characterizing a progress made in the environment as a result of the agent performing the selected action;

generating a respective modified reward for each of one or more particular time steps in the sequence, comprising:

identifying one or more time steps in the sequence of time steps as splice time steps;

for each splice time step:

identifying one or more particular time steps that are more than a threshold number of time steps before the splice time step in the sequence; and

generating a modification for each identified particular time step from a value prediction generated using the data read from the external memory at the splice time step; and

for each identified particular time step, applying the modifications for the identified particular time step to the actual reward for the identified particular time step; and

training, through reinforcement learning, the neural network system using at least the modified rewards for the particular time steps.

2. The method of claim 1 , wherein identifying one or more time steps in the sequence as splice time steps comprises, for each of the one or more read heads, identifying one or more time steps in the sequence of time steps as a splice time step based on a strength with which data was read from the external memory for the read head at the time step.

3. The method of claim 2 ,

wherein reading data from the external memory comprises:

for each read head, generating a read strength parameter that defines how strongly the external memory is read from at the time step, and

wherein identifying one or more time steps in the sequence of time steps as a splice time step based on a strength with which data was read from the external memory for the read head at the time step comprises:

identifying, as splice time steps, one or more time steps for which the read strength parameter for the read head exceeds a threshold value.

4. The method of claim 3 , wherein identifying, as splice time steps, one or more time steps for which the read strength parameter exceeds a threshold value comprises:

identifying a window of consecutive time steps for which the read strength parameter for the read head exceeds the threshold value; and

identifying, as a splice time step, the time step in the window for which the read strength parameter for the read head is the highest.

5. The method of claim 4 , wherein identifying, as splice time steps, one or more time steps for which the read strength parameter exceeds a threshold value comprises:

prior to identifying the window of time steps, setting the read strength parameter for the read head to zero for time steps at which a difference between (i) a time index for the time step and (ii) a time index for the time step that has the largest weight of any time step when reading from the external memory for the read head at the time step is less than a constant value.

6. The method of claim 1 , wherein identifying one or more particular time steps that are more than the threshold number of time steps before the splice time step in the sequence comprises:

determining a half-life value, and

identifying as particular time steps each time step that is more than the half-life value before the splice time step in the sequence.

7. The method of claim 1 , wherein, for each read head and for each time step, reading from the external memory comprises:

assigning a respective weight to the data written to the external memory at each preceding time step; and

generating a read vector that combines the data written to the external memory at each preceding time step in accordance with the respective weights.

8. The method of claim 7 , wherein generating a modification for each identified time step from a value prediction generated using the data read from the external memory at the splice time step comprises:

generating the modification for the identified time step from (i) a future value discount factor, (ii) the weight assigned to the identified time step while reading from the external memory at the splice time step and (iii) the value prediction generated using the data read from the external memory at the splice time step.

9. The method of claim 1 , wherein training, through reinforcement learning, the neural network system using at least the modified rewards for the particular time steps comprises training the neural network system to minimize a loss function that includes a policy gradient loss.

10. The method of claim 9 , wherein the loss function includes a reconstruction loss.

11. The method of claim 9 , wherein the loss function includes a reading reconstruction loss that imposes a regularization cost whenever a read strength is above a threshold value.

12. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of training a neural network system augmented with an external memory and used to control an agent interacting with an environment to perform a specified task, the operations comprising:

causing the agent to perform a task episode in which the agent attempts to perform the specified task, comprising, at each of a sequence of time steps during the task episode:

obtaining an observation characterizing a current state of the environment;

for each of one or more read heads, reading data from the external memory;

writing data to the external memory;

selecting, from the observation and using the neural network system, an action to be performed by the agent in response to the observation, comprising selecting the action using at least (i) data read from the external memory at a preceding time step or (ii) data read from the external memory at the time step;

generating, using the neural network system, a value prediction representing a value of the environment being in the current state to successfully performing the specified task, comprising generating the value prediction using at least (i) data read from the external memory at a preceding time step or (ii) data read from the external memory at the time step;

causing the agent to perform the selected action; and

in response, receiving an actual reward characterizing a progress made in the environment as a result of the agent performing the selected action;

generating a respective modified reward for each of one or more particular time steps in the sequence, comprising:

identifying one or more time steps in the sequence of time steps as splice time steps;

for each splice time step:

identifying one or more particular time steps that are more than a threshold number of time steps before the splice time step in the sequence; and

generating a modification for each identified particular time step from a value prediction generated using the data read from the external memory at the splice time step; and

for each identified particular time step, applying the modifications for the identified particular time step to the actual reward for the identified particular time step; and

training, through reinforcement learning, the neural network system using at least the modified rewards for the particular time steps.

13. A system comprising one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations of training a neural network system augmented with an external memory and used to control an agent interacting with an environment to perform a specified task, the operations comprising:

causing the agent to perform a task episode in which the agent attempts to perform the specified task, comprising, at each of a sequence of time steps during the task episode:

obtaining an observation characterizing a current state of the environment;

for each of one or more read heads, reading data from the external memory;

writing data to the external memory;

selecting, from the observation and using the neural network system, an action to be performed by the agent in response to the observation, comprising selecting the action using at least (i) data read from the external memory at a preceding time step or (ii) data read from the external memory at the time step;

generating, using the neural network system, a value prediction representing a value of the environment being in the current state to successfully performing the specified task, comprising generating the value prediction using at least (i) data read from the external memory at a preceding time step or (ii) data read from the external memory at the time step;

causing the agent to perform the selected action; and

in response, receiving an actual reward characterizing a progress made in the environment as a result of the agent performing the selected action;

generating a respective modified reward for each of one or more particular time steps in the sequence, comprising:

identifying one or more time steps in the sequence of time steps as splice time steps;

for each splice time step:

identifying one or more particular time steps that are more than a threshold number of time steps before the splice time step in the sequence; and

generating a modification for each identified particular time step from a value prediction generated using the data read from the external memory at the splice time step; and

for each identified particular time step, applying the modifications for the identified particular time step to the actual reward for the identified particular time step; and

training, through reinforcement learning, the neural network system using at least the modified rewards for the particular time steps.

14. The system of claim 13 , wherein identifying one or more time steps in the sequence as splice time steps comprises, for each of the one or more read heads, identifying one or more time steps in the sequence of time steps as a splice time step based on a strength with which data was read from the external memory for the read head at the time step.

15. The system of claim 14 ,

wherein reading data from the external memory comprises:

for each read head, generating a read strength parameter that defines how strongly the external memory is read from at the time step, and

wherein identifying one or more time steps in the sequence of time steps as a splice time step based on a strength with which data was read from the external memory for the read head at the time step comprises:

identifying, as splice time steps, one or more time steps for which the read strength parameter for the read head exceeds a threshold value.

16. The system of claim 15 , wherein identifying, as splice time steps, one or more time steps for which the read strength parameter exceeds a threshold value comprises:

identifying a window of consecutive time steps for which the read strength parameter for the read head exceeds the threshold value; and

identifying, as a splice time step, the time step in the window for which the read strength parameter for the read head is the highest.

17. The system of claim 16 , wherein identifying, as splice time steps, one or more time steps for which the read strength parameter exceeds a threshold value comprises:

prior to identifying the window of time steps, setting the read strength parameter for the read head to zero for time steps at which a difference between (i) a time index for the time step and (ii) a time index for the time step that has the largest weight of any time step when reading from the external memory for the read head at the time step is less than a constant value.

18. The system of claim 13 , wherein identifying one or more particular time steps that are more than the threshold number of time steps before the splice time step in the sequence comprises:

determining a half-life value, and

identifying as particular time steps each time step that is more than the half-life value before the splice time step in the sequence.

19. The system of claim 13 , wherein, for each read head and for each time step, reading from the external memory comprises:

assigning a respective weight to the data written to the external memory at each preceding time step; and

generating a read vector that combines the data written to the external memory at each preceding time step in accordance with the respective weights.

20. The system of claim 19 , wherein generating a modification for each identified time step from a value prediction generated using the data read from the external memory at the splice time step comprises:

generating the modification for the identified time step from (i) a future value discount factor, (ii) the weight assigned to the identified time step while reading from the external memory at the splice time step and (iii) the value prediction generated using the data read from the external memory at the splice time step.