Environment prediction using reinforcement learning

1. A method performed by one or more data processing apparatus, the method comprising:

receiving, by the one or more data processing apparatus, one or more observations characterizing states of an environment being interacted with by an agent;

providing, by the one or more data processing apparatus, the one or more observations as input to a state representation neural network, wherein the state representation neural network is configured to:

receive the one or more observations, and

process the one or more observations to generate an internal state representation of a current environment state;

for each of a plurality of internal time steps:

generating, by the one or more data processing apparatus, using a prediction neural network and a value prediction neural network and from an internal state representation for the internal time step: (i) an internal state representation for a next internal time step, (ii) a predicted reward for the next internal time step, and (iii) a value prediction that is an estimate of a future cumulative discounted reward from the next internal time step onwards;

wherein the prediction neural network is configured to, for each of the plurality of internal time steps:

receive the internal state representation for the internal time step; and

process the internal state representation for the internal time step to generate:

an internal state representation for a next internal time step, and a predicted reward for the next internal time step;

wherein the value prediction neural network is configured to, for each of the plurality of internal time steps:

receive the internal state representation for the internal time step, and

process the internal state representation for the internal time step to generate a value prediction that is an estimate of a future cumulative discounted reward from the next internal time step onwards; and

determining, by the one or more data processing apparatus, an aggregate reward from the predicted rewards and the value predictions for the internal time steps, wherein the aggregate reward is an estimate of an outcome associated with the states of the environment characterized by the observations.

2. The method of claim 1 , further comprising:

providing the aggregate reward as an estimate of the outcome associated with the states of the environment characterized by the observations.

3. The method of claim 1 , wherein the prediction neural network is further configured to generate a predicted discount factor for the next internal time step, and further comprising using the predicted discount factors for the internal time steps in determining the aggregate reward.

4. The method of claim 1 , wherein the state representation neural network comprises a recurrent neural network.

5. The method of claim 3 , further comprising:

for each internal time step,

processing, by the one or more data processing apparatus, an internal state representation for the internal time step using a lambda neural network to generate a lambda factor for the next internal time step;

determining, by the one or more data processing apparatus, a respective k-step return for each internal time step by combining the predicted reward and the predicted discount factor for each of a first k internal time steps and the value prediction for a k-th internal time step; and

using the lambda factors to determine weights for the k-step returns in determining the aggregate reward.

6. The method of claim 1 , wherein the state representation neural network comprises a feedforward neural network.

7. The method of claim 1 , wherein the prediction neural network comprises a recurrent neural network.

8. The method of claim 1 , wherein the prediction neural network comprises a feedforward neural network that has different parameter values at each of the plurality of internal time steps.

9. A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

receiving, by the one or more computers, one or more observations characterizing states of an environment being interacted with by an agent;

providing, by the one or more computers, the one or more observations as input to a state representation neural network, wherein the state representation neural network is configured to:

receive the one or more observations, and

process the one or more observations to generate an internal state representation of a current environment state;

for each of a plurality of internal time steps:

generating, by the one or more computers, using a prediction neural network and a value prediction neural network and from an internal state representation for the internal time step: (i) an internal state representation for a next internal time step, (ii) a predicted reward for the next internal time step, and (iii) a value prediction that is an estimate of a future cumulative discounted reward from the next internal time step onwards;

wherein the prediction neural network is configured to, for each of the plurality of internal time steps:

receive the internal state representation for the internal time step; and

process the internal state representation for the internal time step to generate:

an internal state representation for a next internal time step, and

a predicted reward for the next internal time step;

wherein the value prediction neural network is configured to, for each of the plurality of internal time steps:

receive the internal state representation for the internal time step, and

process the internal state representation for the internal time step to generate a value prediction that is an estimate of a future cumulative discounted reward from the next internal time step onwards; and

determining, by the one or more computers, an aggregate reward from the predicted rewards and the value predictions for the internal time steps, wherein the aggregate reward is an estimate of an outcome associated with the states of the environment characterized by the observations.

10. The system of claim 9 , wherein the operations further comprise:

providing the aggregate reward as an estimate of the outcome associated with the states of the environment characterized by the observations.

11. The system of claim 9 , wherein the prediction neural network is further configured to generate a predicted discount factor for the next internal time step, and wherein the operations further comprise using the predicted discount factors for the internal time steps in determining the aggregate reward.

12. The system of claim 9 , wherein the state representation neural network comprises a recurrent neural network.

13. The system of claim 11 , wherein the operations further comprise:

for each internal time step,

processing, by the one or more data processing apparatus, an internal state representation for the internal time step using a lambda neural network to generate a lambda factor for the next internal time step;

determining, by the one or more computers, a respective k-step return for each internal time step by combining the predicted reward and the predicted discount factor for each of a first k internal time steps and the value prediction for a k-th internal time step; and

using the lambda factors to determine weights for the k-step returns in determining the aggregate reward.

14. The system of claim 9 , wherein the state representation neural network comprises a feedforward neural network.

15. The system of claim 9 , wherein the prediction neural network comprises a recurrent neural network.

16. The system of claim 9 , wherein the prediction neural network comprises a feedforward neural network that has different parameter values at each of the plurality of internal time steps.

17. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving, by the one or more computers, one or more observations characterizing states of an environment being interacted with by an agent;

providing, by the one or more computers, the one or more observations as input to a state representation neural network, wherein the state representation neural network is configured to:

receive the one or more observations, and

process the one or more observations to generate an internal state representation of a current environment state;

for each of a plurality of internal time steps:

generating, by the one or more computers, using a prediction neural network and a value prediction neural network and from an internal state representation for the internal time step: (i) an internal state representation for a next internal time step, (ii) a predicted reward for the next internal time step, and (iii) a value prediction that is an estimate of a future cumulative discounted reward from the next internal time step onwards;

wherein the prediction neural network is configured to, for each of the plurality of internal time steps:

receive the internal state representation for the internal time step; and

process the internal state representation for the internal time step to generate:

an internal state representation for a next internal time step, and

a predicted reward for the next internal time step;

wherein the value prediction neural network is configured to, for each of the plurality of internal time steps:

receive the internal state representation for the internal time step, and

process the internal state representation for the internal time step to generate a value prediction that is an estimate of a future cumulative discounted reward from the next internal time step onwards; and

determining, by the one or more computers, an aggregate reward from the predicted rewards and the value predictions for the internal time steps, wherein the aggregate reward is an estimate of an outcome associated with the states of the environment characterized by the observations.

18. The one or more non-transitory computer storage media of claim 17 , wherein the operations further comprise:

providing the aggregate reward as an estimate of the outcome associated with the states of the environment characterized by the observations.

19. The one or more non-transitory computer storage media of claim 17 , wherein the prediction neural network is further configured to generate a predicted discount factor for the next internal time step, and wherein the operations further comprise using the predicted discount factors for the internal time steps in determining the aggregate reward.

20. The one or more non-transitory computer storage media of claim 19 , wherein the operations further comprise:

for each internal time step,

processing, by the one or more data processing apparatus, an internal state representation for the internal time step using a lambda neural network to generate a lambda factor for the next internal time step;

determining, by the one or more computers, a respective k-step return for each internal time step by combining the predicted reward and the predicted discount factor for each of a first k internal time steps and the value prediction for a k-th internal time step; and

using the lambda factors to determine weights for the k-step returns in determining the aggregate reward.