Reinforcement learning using meta-learned intrinsic rewards
There is described methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a reinforcement learning system. The reinforcement learning system comprises an agent configured to perform actions based upon a policy and an intrinsic reward system configured to generate intrinsic reward values for the agent based upon the actions taken by the agent. The method comprises training the reinforcement learning system based upon a plurality of tasks. The training comprises updating the agent's policy based upon the intrinsic reward values generated by the intrinsic reward system and updating the intrinsic reward system based upon an extrinsic reward value obtained based upon the task being performed by the agent. The training further comprises re-initializing the agent's policy when an expiration criterion associated with the agent is met.
1. A method of training a reinforcement learning system, the reinforcement learning system comprising an agent configured to perform actions based upon a policy and an intrinsic reward system configured to generate intrinsic reward values for the agent based upon the actions taken by the agent, the method comprising:
training the reinforcement learning system by updating the agent's policy based upon, for each of a plurality of task episodes performed by the agent and each corresponding to a respective task, a first objective that measures an expected return accumulated within the task episode, the training comprising, for each of the plurality of task episodes:
at each of a plurality of time steps within the task episode, determining a respective intrinsic reward value for the time step using the intrinsic reward system conditioned on a history of the agent at earlier time steps within the task episode and at time steps during previous task episodes across a lifetime of the agent, wherein the lifetime of the agent comprises the plurality of task episodes; and
updating the intrinsic reward system based upon a second objective that measures, for a particular time step within a particular one of the plurality of task episodes, an expected return accumulated across a remainder of the lifetime of the agent subsequent to the particular time step, wherein the expected return accumulated across the remainder of the lifetime of the agent is determined based on extrinsic reward values at one or more subsequent time steps and on an approximation of a lifetime extrinsic reward for subsequent task episodes during the remainder of the lifetime of the agent generated by a lifetime return function.
2. A method according to claim 1 , wherein the history of the agent comprises at least one of the following: observations of the environment, the agent's actions, the extrinsic rewards, or an indication of whether a state is terminal.
3. A method according to claim 1 , wherein updating the agent's policy comprises repeatedly updating the agent's policy for a plurality of updates on a same task and wherein updating the intrinsic reward system follows the repeated updating of the agent's policy.
4. A method according to claim 1 , wherein updating the intrinsic reward system comprises computing a policy gradient based upon maximizing the expected return accumulated over the lifetime of the agent.
5. A method according to claim 4 , wherein the policy gradient based upon maximizing the expected return accumulated over the lifetime of the agent is computed based upon the lifetime return function, wherein the return function comprises a stochastic estimate of a lifetime return value.
6. A method according to claim 5 , wherein the stochastic estimate of the lifetime return value is generated by a deep recurrent neural network.
7. A method according to claim 6 , wherein the method further comprises updating one or more parameters of the lifetime return function.
8. A method according to claim 7 , wherein updating the parameters of the lifetime return function is based upon temporal difference learning.
9. A method according to claim 8 , wherein the parameters of the lifetime return function are maintained when an expiration criterion for re-initializing the policy of the agent is met.
10. A method according to claim 4 , wherein the policy gradient based upon maximizing the expected return accumulated over the lifetime of the agent is further based upon accumulating a gradient computed based upon each update of the agent's policy.
11. A method according to claim 1 , wherein updating the agent's policy comprises, for each task episode, computing a policy gradient based upon maximizing the expected return accumulated within the task episode.
12. A method according to claim 1 , wherein the intrinsic reward system comprises a recurrent neural network configured to generate the respective intrinsic reward values.
13. A method according to claim 1 , further comprising:
re-initializing the agent's policy when an expiration criterion is met, wherein the expiration criterion is based upon a number of training episodes completed by the agent.
14. A method according to claim 1 , wherein the agent's policy is provided by a policy neural network.
15. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a reinforcement learning system, the reinforcement learning system comprising an agent configured to perform actions based upon a policy and an intrinsic reward system configured to generate intrinsic reward values for the agent based upon the actions taken by the agent, the operations comprising:
training the reinforcement learning system by updating the agent's policy based upon, for each of a plurality of task episodes performed by the agent and each corresponding to a respective task, a first objective that measures an expected return accumulated within the task episode, the training comprising, for each of the plurality of task episodes:
at each of a plurality of time steps within the task episode, determining a respective intrinsic reward value for the time step using the intrinsic reward system conditioned on a history of the agent at earlier time steps within the task episode and at time steps during previous task episodes across a lifetime of the agent, wherein the lifetime of the agent comprises the plurality of task episodes; and
updating the intrinsic reward system based upon a second objective that measures, for a particular time step within a particular one of the plurality of task episodes, an expected return accumulated across a remainder of the lifetime of the agent subsequent to the particular time step, wherein the expected return accumulated across the remainder of the lifetime of the agent is determined based on extrinsic reward values at one or more subsequent time steps and on an approximation of a lifetime extrinsic reward for subsequent task episodes during the remainder of the lifetime of the agent generated by a lifetime return function.
16. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a reinforcement learning system, the reinforcement learning system comprising an agent configured to perform actions based upon a policy and an intrinsic reward system configured to generate intrinsic reward values for the agent based upon the actions taken by the agent, the operations comprising:
training the reinforcement learning system by updating the agent's policy based upon, for each of a plurality of task episodes performed by the agent and each corresponding to a respective task, a first objective that measures an expected return accumulated within the task episode, the training comprising, for each of the plurality of task episodes:
at each of a plurality of time steps within the task episode, determining a respective intrinsic reward value for the time step using the intrinsic reward system conditioned on a history of the agent at earlier time steps within the task episode and at time steps during previous task episodes across a lifetime of the agent, wherein the lifetime of the agent comprises the plurality of task episodes; and
updating the intrinsic reward system based upon a second objective that measures, for a particular time step within a particular one of the plurality of task episodes, an expected return accumulated across a remainder of the lifetime of the agent subsequent to the particular time step, wherein the expected return accumulated across the remainder of the lifetime of the agent is determined based on extrinsic reward values at one or more subsequent time steps and on an approximation of a lifetime extrinsic reward for subsequent task episodes during the remainder of the lifetime of the agent generated by a lifetime return function.
17. A reinforcement learning system, the system comprising:
an agent configured to perform actions based upon a policy, and
an intrinsic reward system implemented by one or more computers and configured to generate intrinsic reward values for the agent based upon the actions taken by the agent; and
wherein the reinforcement learning system is trained by performing operations comprising:
training the reinforcement learning system by updating the agent's policy based upon, for each of a plurality of task episodes performed by the agent and each corresponding to a respective task, a first objective that measures an expected return accumulated within the task episode, the training comprising, for each of the plurality of task episodes:
at each of a plurality of time steps within the task episode, determining a respective intrinsic reward value for the time step using the intrinsic reward system conditioned on a history of the agent at earlier time steps within the task episode and at time steps during previous task episodes across a lifetime of the agent, wherein the lifetime of the agent comprises the plurality of task episodes; and
updating the intrinsic reward system based upon a second objective that measures, for a particular time step within a particular one of the plurality of task episodes, an expected return accumulated across a remainder of the lifetime of the agent subsequent to the particular time step, wherein the expected return accumulated across the remainder of the lifetime of the agent is determined based on extrinsic reward values at one or more subsequent time steps and on an approximation of a lifetime extrinsic reward for subsequent task episodes during the remainder of the lifetime of the agent generated by a lifetime return function.
18. A system according to claim 15 , wherein the history of the agent comprises at least one of the following: observations of the environment, the agent's actions, the extrinsic rewards, or an indication of whether a state is terminal.
19. A system according to claim 15 , wherein updating the agent's policy comprises repeatedly updating the agent's policy for a plurality of updates on a same task and wherein updating the intrinsic reward system follows the repeated updating of the agent's policy.
20. A system according to claim 15 , wherein updating the intrinsic reward system comprises computing a policy gradient based upon maximizing the expected return accumulated over the lifetime of the agent.