Reinforcement learning with auxiliary tasks

1. A method performed by one or more data processing apparatus, the method comprising:

training an action selection policy neural network using a first reinforcement learning technique,

wherein the action selection policy neural network has a plurality of network parameters and is used in selecting actions to be performed by an agent interacting with an environment,

wherein the action selection policy neural network is configured to receive an input comprising an observation input and to process the input in accordance with the network parameters to generate an action selection policy output, and

wherein training the action selection policy neural network comprises adjusting values of the action selection policy network parameters;

during the training of the action selection neural network using the first reinforcement learning technique:

training a reward prediction neural network on interactions of the agent with the environment during the training of the action selection neural network,

wherein the reward prediction neural network has reward prediction parameters and is configured to:

receive a plurality of intermediate outputs generated by the action selection policy neural network, wherein the plurality of intermediate outputs are generated by one or more hidden layers of the action selection policy neural network in response to processing a sequence of multiple observation images that result from interactions of the agent with the environment, and

process the plurality of intermediate outputs, generated by the hidden layers of the action selection policy neural network in response to processing the sequence of multiple observation images, in accordance with the reward prediction parameters to generate a predicted reward that is an estimate of a reward that will be received with a next observation image that follows a last observation image in the sequence; and

wherein training the reward prediction neural network comprises:

determining gradients based on predicted rewards generated by the reward prediction neural network; and

adjusting values of the reward prediction parameters and the action selection policy network parameters using the gradients.

2. The method of claim 1 , wherein training the reward prediction neural network comprises:

receiving an actual reward received with the next observation image; and

training the reward prediction neural network to decrease a loss between the actual reward and the estimated reward.

3. The method of claim 2 , wherein training the reward prediction neural network to decrease a loss between the actual reward and the estimated reward comprises:

backpropagating gradients computed into the action selection policy neural network to adjust the values of the action selection policy parameters.

4. The method of claim 1 , wherein training the reward prediction neural network comprises sampling sequences of observations from a replay memory.

5. The method of claim 4 , wherein sampling sequences of observations comprises:

sampling sequences for which the actual reward received with the observation following the last observation in the sequence is non-zero with a higher probability than sequences for which the actual reward received with the observation following the last observation in the sequence is zero.

6. The method of claim 4 , wherein the first reinforcement learning technique is an actor-critic reinforcement learning technique, wherein the action selection policy output comprises value estimates and Q-values, and wherein the method further comprises:

resampling recent sequences from the replay memory; and

performing extra value function regression on the value estimates for the observation images in the resampled recent sequences.

7. The method of claim 1 ,

wherein the action selection policy neural network comprises:

a convolutional encoder neural network that generates an encoded representation of the input,

an intermediate neural network that processes the encoded representation of the input to generate an intermediate representation; and

an output neural network that processes the intermediate representation to generate the action selection policy output.

8. The method of claim 7 , wherein the intermediate neural network is a recurrent neural network and the plurality of intermediate outputs generated by the action selection policy neural network are the intermediate representation for a last observation image in the sequence.

9. The method of claim 7 , wherein the plurality of intermediate outputs generated by the action selection policy neural network are the encoded representations for the observation images in the sequence.

10. A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

training an action selection policy neural network using a first reinforcement learning technique,

wherein the action selection policy neural network has a plurality of network parameters and is used in selecting actions to be performed by an agent interacting with an environment,

wherein the action selection policy neural network is configured to receive an input comprising an observation input and to process the input in accordance with the network parameters to generate an action selection policy output, and

wherein training the action selection policy neural network comprises adjusting values of the action selection policy network parameters;

during the training of the action selection neural network using the first reinforcement learning technique:

training a reward prediction neural network on interactions of the agent with the environment during the training of the action selection neural network,

wherein the reward prediction neural network has reward prediction parameters and is configured to:

receive a plurality of intermediate outputs generated by the action selection policy neural network, wherein the plurality of intermediate outputs are generated by one or more hidden layers of the action selection policy neural network in response to processing a sequence of multiple observation images that result from interactions of the agent with the environment, and

process the plurality of intermediate outputs, generated by the hidden layers of the action selection policy neural network in response to processing the sequence of multiple observation images, in accordance with the reward prediction parameters to generate a predicted reward that is an estimate of a reward that will be received with a next observation image that follows a last observation image in the sequence; and

wherein training the reward prediction neural network comprises:

determining gradients based on predicted rewards generated by the reward prediction neural network; and

adjusting values of the reward prediction parameters and the action selection policy network parameters using the gradients.

11. The system of claim 10 , wherein training the reward prediction neural network comprises:

receiving an actual reward received with the next observation image; and

training the reward prediction neural network to decrease a loss between the actual reward and the estimated reward.

12. The system of claim 11 , wherein training the reward prediction neural network to decrease a loss between the actual reward and the estimated reward comprises:

backpropagating gradients computed into the action selection policy neural network to adjust the values of the action selection policy parameters.

13. The system of claim 10 , wherein training the reward prediction neural network comprises sampling sequences of observations from a replay memory.

14. The system of claim 13 , wherein sampling sequences of observations comprises:

sampling sequences for which the actual reward received with the observation following the last observation in the sequence is non-zero with a higher probability than sequences for which the actual reward received with the observation following the last observation in the sequence is zero.

15. The system of claim 13 , wherein the first reinforcement learning technique is an actor-critic reinforcement learning technique, wherein the action selection policy output comprises value estimates and Q-values, and wherein the operations further comprise:

resampling recent sequences from the replay memory; and

performing extra value function regression on the value estimates for the observation images in the resampled recent sequences.

16. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

training an action selection policy neural network using a first reinforcement learning technique,

wherein the action selection policy neural network has a plurality of network parameters and is used in selecting actions to be performed by an agent interacting with an environment,

wherein the action selection policy neural network is configured to receive an input comprising an observation input and to process the input in accordance with the network parameters to generate an action selection policy output, and

wherein training the action selection policy neural network comprises adjusting values of the action selection policy network parameters;

during the training of the action selection neural network using the first reinforcement learning technique:

training a reward prediction neural network on interactions of the agent with the environment during the training of the action selection neural network,

wherein the reward prediction neural network has reward prediction parameters and is configured to:

receive a plurality of intermediate outputs generated by the action selection policy neural network, wherein the plurality of intermediate outputs are generated by one or more hidden layers of the action selection policy neural network in response to processing a sequence of multiple observation images that result from interactions of the agent with the environment, and

process the plurality of intermediate outputs, generated by the hidden layers of the action selection policy neural network in response to processing the sequence of multiple observation images, in accordance with the reward prediction parameters to generate a predicted reward that is an estimate of a reward that will be received with a next observation image that follows a last observation image in the sequence; and

wherein training the reward prediction neural network comprises:

determining gradients based on predicted rewards generated by the reward prediction neural network; and

adjusting values of the reward prediction parameters and the action selection policy network parameters using the gradients.

17. The non-transitory computer storage media of claim 16 , wherein training the reward prediction neural network comprises:

receiving an actual reward received with the next observation image; and

training the reward prediction neural network to decrease a loss between the actual reward and the estimated reward.

18. The non-transitory computer storage media of claim 17 , wherein training the reward prediction neural network to decrease a loss between the actual reward and the estimated reward comprises:

backpropagating gradients computed into the action selection policy neural network to adjust the values of the action selection policy parameters.

19. The non-transitory computer storage media of claim 16 , wherein training the reward prediction neural network comprises sampling sequences of observations from a replay memory.

20. The non-transitory computer storage media of claim 19 , wherein the first reinforcement learning technique is an actor-critic reinforcement learning technique, wherein the action selection policy output comprises value estimates and Q-values, and wherein the operations further comprise:

resampling recent sequences from the replay memory; and

performing extra value function regression on the value estimates for the observation images in the resampled recent sequences.