IP Library › Granted Patent US 12,293,283
Granted Patent B2
US 12,293,283 · App. 17/033,410 · Granted May 6, 2025

Reinforcement learning using meta-learned intrinsic rewards

Inventors: Zeyu Zheng (Ann Arbor, MI); Junhyuk Oh (London, GB); Satinder Singh Baveja (Ann Arbor, MI)
Assignee: DeepMind Technologies Limited
G06N3/08G06N3/044G06N3/045G06N3/084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,293,283
App. No.
17/033,410
Granted
May 6, 2025
Kind
B2
Abstract

There is described methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a reinforcement learning system. The reinforcement learning system comprises an agent configured to perform actions based upon a policy and an intrinsic reward system configured to generate intrinsic reward values for the agent based upon the actions taken by the agent. The method comprises training the reinforcement learning system based upon a plurality of tasks. The training comprises updating the agent's policy based upon the intrinsic reward values generated by the intrinsic reward system and updating the intrinsic reward system based upon an extrinsic reward value obtained based upon the task being performed by the agent. The training further comprises re-initializing the agent's policy when an expiration criterion associated with the agent is met.

Claims (36)

1. A method of training a reinforcement learning system, the reinforcement learning system comprising an agent configured to perform actions based upon a policy and an intrinsic reward system configured to generate intrinsic reward values for the agent based upon the actions taken by the agent, the method comprising:

training the reinforcement learning system by updating the agent's policy based upon, for each of a plurality of task episodes performed by the agent and each corresponding to a respective task, a first objective that measures an expected return accumulated within the task episode, the training comprising, for each of the plurality of task episodes:

at each of a plurality of time steps within the task episode, determining a respective intrinsic reward value for the time step using the intrinsic reward system conditioned on a history of the agent at earlier time steps within the task episode and at time steps during previous task episodes across a lifetime of the agent, wherein the lifetime of the agent comprises the plurality of task episodes; and

updating the intrinsic reward system based upon a second objective that measures, for a particular time step within a particular one of the plurality of task episodes, an expected return accumulated across a remainder of the lifetime of the agent subsequent to the particular time step, wherein the expected return accumulated across the remainder of the lifetime of the agent is determined based on extrinsic reward values at one or more subsequent time steps and on an approximation of a lifetime extrinsic reward for subsequent task episodes during the remainder of the lifetime of the agent generated by a lifetime return function.

2. A method according to claim 1 , wherein the history of the agent comprises at least one of the following: observations of the environment, the agent's actions, the extrinsic rewards, or an indication of whether a state is terminal.

3. A method according to claim 1 , wherein updating the agent's policy comprises repeatedly updating the agent's policy for a plurality of updates on a same task and wherein updating the intrinsic reward system follows the repeated updating of the agent's policy.

4. A method according to claim 1 , wherein updating the intrinsic reward system comprises computing a policy gradient based upon maximizing the expected return accumulated over the lifetime of the agent.

5. A method according to claim 4 , wherein the policy gradient based upon maximizing the expected return accumulated over the lifetime of the agent is computed based upon the lifetime return function, wherein the return function comprises a stochastic estimate of a lifetime return value.

6. A method according to claim 5 , wherein the stochastic estimate of the lifetime return value is generated by a deep recurrent neural network.

7. A method according to claim 6 , wherein the method further comprises updating one or more parameters of the lifetime return function.

8. A method according to claim 7 , wherein updating the parameters of the lifetime return function is based upon temporal difference learning.

9. A method according to claim 8 , wherein the parameters of the lifetime return function are maintained when an expiration criterion for re-initializing the policy of the agent is met.

10. A method according to claim 4 , wherein the policy gradient based upon maximizing the expected return accumulated over the lifetime of the agent is further based upon accumulating a gradient computed based upon each update of the agent's policy.

11. A method according to claim 1 , wherein updating the agent's policy comprises, for each task episode, computing a policy gradient based upon maximizing the expected return accumulated within the task episode.

12. A method according to claim 1 , wherein the intrinsic reward system comprises a recurrent neural network configured to generate the respective intrinsic reward values.

13. A method according to claim 1 , further comprising:

re-initializing the agent's policy when an expiration criterion is met, wherein the expiration criterion is based upon a number of training episodes completed by the agent.

14. A method according to claim 1 , wherein the agent's policy is provided by a policy neural network.

15. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a reinforcement learning system, the reinforcement learning system comprising an agent configured to perform actions based upon a policy and an intrinsic reward system configured to generate intrinsic reward values for the agent based upon the actions taken by the agent, the operations comprising:

training the reinforcement learning system by updating the agent's policy based upon, for each of a plurality of task episodes performed by the agent and each corresponding to a respective task, a first objective that measures an expected return accumulated within the task episode, the training comprising, for each of the plurality of task episodes:

at each of a plurality of time steps within the task episode, determining a respective intrinsic reward value for the time step using the intrinsic reward system conditioned on a history of the agent at earlier time steps within the task episode and at time steps during previous task episodes across a lifetime of the agent, wherein the lifetime of the agent comprises the plurality of task episodes; and

updating the intrinsic reward system based upon a second objective that measures, for a particular time step within a particular one of the plurality of task episodes, an expected return accumulated across a remainder of the lifetime of the agent subsequent to the particular time step, wherein the expected return accumulated across the remainder of the lifetime of the agent is determined based on extrinsic reward values at one or more subsequent time steps and on an approximation of a lifetime extrinsic reward for subsequent task episodes during the remainder of the lifetime of the agent generated by a lifetime return function.

16. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a reinforcement learning system, the reinforcement learning system comprising an agent configured to perform actions based upon a policy and an intrinsic reward system configured to generate intrinsic reward values for the agent based upon the actions taken by the agent, the operations comprising:

training the reinforcement learning system by updating the agent's policy based upon, for each of a plurality of task episodes performed by the agent and each corresponding to a respective task, a first objective that measures an expected return accumulated within the task episode, the training comprising, for each of the plurality of task episodes:

at each of a plurality of time steps within the task episode, determining a respective intrinsic reward value for the time step using the intrinsic reward system conditioned on a history of the agent at earlier time steps within the task episode and at time steps during previous task episodes across a lifetime of the agent, wherein the lifetime of the agent comprises the plurality of task episodes; and

updating the intrinsic reward system based upon a second objective that measures, for a particular time step within a particular one of the plurality of task episodes, an expected return accumulated across a remainder of the lifetime of the agent subsequent to the particular time step, wherein the expected return accumulated across the remainder of the lifetime of the agent is determined based on extrinsic reward values at one or more subsequent time steps and on an approximation of a lifetime extrinsic reward for subsequent task episodes during the remainder of the lifetime of the agent generated by a lifetime return function.

17. A reinforcement learning system, the system comprising:

an agent configured to perform actions based upon a policy, and

an intrinsic reward system implemented by one or more computers and configured to generate intrinsic reward values for the agent based upon the actions taken by the agent; and

wherein the reinforcement learning system is trained by performing operations comprising:

training the reinforcement learning system by updating the agent's policy based upon, for each of a plurality of task episodes performed by the agent and each corresponding to a respective task, a first objective that measures an expected return accumulated within the task episode, the training comprising, for each of the plurality of task episodes:

at each of a plurality of time steps within the task episode, determining a respective intrinsic reward value for the time step using the intrinsic reward system conditioned on a history of the agent at earlier time steps within the task episode and at time steps during previous task episodes across a lifetime of the agent, wherein the lifetime of the agent comprises the plurality of task episodes; and

updating the intrinsic reward system based upon a second objective that measures, for a particular time step within a particular one of the plurality of task episodes, an expected return accumulated across a remainder of the lifetime of the agent subsequent to the particular time step, wherein the expected return accumulated across the remainder of the lifetime of the agent is determined based on extrinsic reward values at one or more subsequent time steps and on an approximation of a lifetime extrinsic reward for subsequent task episodes during the remainder of the lifetime of the agent generated by a lifetime return function.

18. A system according to claim 15 , wherein the history of the agent comprises at least one of the following: observations of the environment, the agent's actions, the extrinsic rewards, or an indication of whether a state is terminal.

19. A system according to claim 15 , wherein updating the agent's policy comprises repeatedly updating the agent's policy for a plurality of updates on a same task and wherein updating the intrinsic reward system follows the repeated updating of the agent's policy.

20. A system according to claim 15 , wherein updating the intrinsic reward system comprises computing a policy gradient based upon maximizing the expected return accumulated over the lifetime of the agent.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 6, 2025
From: DEEPMIND TECHNOLOGIES LIMITED
To: GDM HOLDING LLC
Reel/Frame 071498/0210 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 19, 2020
From: ZHENG, ZEYU; OH, JUNHYUK; BAVEJA, SATINDER SINGH
To: DEEPMIND TECHNOLOGIES LIMITED
Reel/Frame 054098/0522 →
Continuity (2)
Provisional Application 62905964 · Sep 25, 2019
Related Publication 20210089910A1 · Mar 25, 2021
References Cited (60)
US 20190279081A1 · Pham · 2019 [cited by examiner]
WO WO2018153807A1 · 2018 [cited by examiner]
WO WO2018156891A1 · 2018 [cited by examiner]
Şimşek, Ö., & Barto, A. G. (2006). An intrinsic reward mechanism for efficient exploration. An Intrinsic Reward Mechanism for Efficient Exploration. https://doi.org/10.1145/1143844.1143949 (Year: 2006). [cited by examiner]
Singh, S., Lewis, R. L., Barto, A. G., & Sorg, J. (2010). Intrinsically Motivated Reinforcement Learning: an Evolutionary perspective. IEEE Transactions on Autonomous Mental Development, 2(2), 70-82. https://doi.org/10.… [cited by examiner]
Kulkarni, T., Narasimhan, K., Saeedi, A., & Tenenbaum, J. B. (2016). Hierarchical deep reinforcement learning: integrating temporal abstraction and intrinsic motivation. arXiv (Cornell University). https://arxiv.org/pdf… [cited by examiner]
Zheng, Z., Oh, J., & Singh, S. (2018). On learning intrinsic rewards for policy gradient methods. arXiv (Cornell University), 31, 4644-4654. https://arxiv.org/pdf/1804.06459.pdf (Year: 2018). [cited by examiner]
Kahn, Gregory, et al. “Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation.” 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018, https://doi.org/1… [cited by examiner]
Savinov, et al., “Episodic curiosity through reachability.” (Year: 2018). [cited by examiner]
Ostrovski, et al., “Count-based exploration with neural density models.” (Year: 2017). [cited by examiner]
Montúfar et al., “Information theoretically aided reinforcement learning for embodied agents.” (Year: 2016). [cited by examiner]
AssaAbloy.com [online], “Faulty Reward Functions in the Wild,” Dec. 21, 2016, retrieved on Jan. 29, 2021, retrieved from URL<https://openai.com/blog/faulty-reward-functions/>, 4 pages. [cited by applicant]
Bahdanau et al., “Learning to understand goal specifications by modelling reward,” CoRR, Jun. 2018, arxiv.org/abs/1806.01946, 19 pages. [cited by applicant]
Bechtle et al., “Meta-learning via learned loss,” CoRR, Jun. 2019, arXiv:1906.05374, 9 pages. [cited by applicant]
Bellemare et al., “Unifying count-based exploration and intrinsic motivation,” CoRR, Nov. 2016, arxiv.org/abs/1606.01868, 20 pages. [cited by applicant]
Cohen et al., “Should I stay or should I go? How the human brain manages the tradeoff between exploitation and exploration,” Philosophical Transactions of the Royal Society B: Biological Sciences, May 2007, 362(1481):93… [cited by applicant]
Duan et al., “One-shot imitation learning,” CoRR, Mar. 2017, arxiv.org/abs/1703.07326, 27 pages. [cited by applicant]
Duan et al., “RL2: Fast reinforcement learning via slow reinforcement learning,” CoRR, Nov. 2016, arXiv:1611.02779, 14 pages. [cited by applicant]
Dubey et al., “Reconciling novelty and complexity through a rational analysis of curiosity,” Psychological Review, 2020, 127(3):455-476. [cited by applicant]
Finn et al., “Model-agnostic metalearning for fast adaptation of deep networks,” Proceedings of the 34th International Conference on Machine Learning, Jul. 2017, 70:1126-1135. [cited by applicant]
Finn et al., “Oneshot visual imitation learning via meta-learning,” Proceedings of the 1st Annual Conference on Robot Learning, Oct. 2018, 78:357-368. [cited by applicant]
Gittins, “Bandit processes and dynamic allocation indices,” Journal of the Royal Statistical Society: Series B (Methodological), Jan. 1979, 41(2):148-177. [cited by applicant]
Gordon et al., “Reinforcement active learning hierarchical loops,” The 2011 International Joint Conference on Neural Networks, Jul. 2011, pp. 3008-3015. [cited by applicant]
Goyal et al., “Infobot: Transfer and exploration via the information bottleneck,” CoRR, Jan. 2019, arxiv.org/abs/1901.10902, 21 pages. [cited by applicant]
Guo et al., “Deep learning for reward design to improve monte carlo tree search in atari games,” CoRR, Apr. 2016, arxiv.org/abs/1604.07095, 7 pages. [cited by applicant]
Harada et al., “Policy invariance under reward transformations: Theory and application to reward shaping,” Proceedings of the Sixteenth International Conference on Machine Learning, 1999, 10 pages. [cited by applicant]
Harutyunyan et al., “Expressing arbitrary reward functions as potential-based advice,” In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, Feb. 2015, 29(1):2652-2658. [cited by applicant]
Itti et al., “Bayesian surprise attracts human attention,” Vision Research, Jun. 2009, 49(10):1295-1306. [cited by applicant]
Kirsch et al., “Improving generalization in meta reinforcement learning using learned objectives,” CoRR, Oct. 2019, arxiv.org/abs/1910.04098, 21 pages. [cited by applicant]
Learning to learn: Introduction and overview, Springer, 1998, Chapter 1, 15 pages. [cited by applicant]
Linke et al., “Adapting behaviour via intrinsic reward: A survey and empirical study,” Jair, Dec. 2020, 69:1287-332. [cited by applicant]
Metz et al., “Meta-learning update rules for unsupervised representation learning,” CoRR, Mar. 2018, arxiv.org/abs/1804.00222, 27 pages. [cited by applicant]
Mirolli et al., “Functions and mechanisms of intrinsic motivations,” Intrinsically Motivated Learning in Natural and Artificial Systems, Nov. 2012, pp. 49-72. [cited by applicant]
Nagabandi et al., “Learning to adapt in dynamic, real-world environments through meta-reinforcement learning,” CoRR, Mar. 2018, arxiv.org/abs/1803.11347, 17 pages. [cited by applicant]
Ostrovski et al., “Count-based exploration with neural density models,” Proceedings of the 34th International Conference on Machine Learning, Jul. 2017, 70:2721-2730. [cited by applicant]
Oudeyer et al., “Intrinsic motivation systems for autonomous mental development,” IEEE transactions on evolutionary computation, Apr. 2007, 11(2):265-286. [cited by applicant]
Pathak et al., “Curiosity-driven exploration by self-supervised prediction,” Proceedings of the 34th International Conference on Machine Learning, Jul. 2017, 70:2778-2787. [cited by applicant]
Poupart et al., “An analytic solution to discrete bayesian reinforcement learning,” Proceedings of the 23rd International Conference on Machine Learning, Jun. 2006, pp. 697-704. [cited by applicant]
Randlov et al., “Learning to drive a bicycle using reinforcement learning and shaping,” Proceedings of the Fifteenth International Conference on Machine Learning, Jul. 1998, 98:463-471. [cited by applicant]
Schlegel et al., “Discovery of predictive representations with a network of general value functions,” ICLR, Feb. 2018, 14 pages. [cited by applicant]
Schmidhuber et al., “Curious model-building control systems,” In Proc. international joint conference on neural networks, 1991, 2:1458-1463. [cited by applicant]
Schmidhuber et al., “Simple principles of metalearning,” Technical Report IDSIA, Jun. 1996, 23 pages. [cited by applicant]
Schmidhuber, “A possibility for implementing curiosity and boredom in model-building neural controllers,” Proc. of the international conference on simulation of adaptive behavior: From animals to animats, 1991, pp. 222-… [cited by applicant]
Singh et al., “Intrinsically motivated reinforcement learning: An evolutionary perspective,” IEEE Transactions on Autonomous Mental Development, Jun. 2010, 2(2):70-82. [cited by applicant]
Singh et al., “Where do rewards come from?,” Proceedings of the annual conference of the cognitive science society, Aug. 2009, pp. 2601-2606. [cited by applicant]
Sorg et al., “Reward design via online gradient ascent,” Advances in Neural Information Processing Systems, 2010, pp. 2190-2198. [cited by applicant]
Stadie et al., “The importance of sampling inmeta-reinforcement learning,” Advances in Neural Information Processing Systems, Dec. 2018, pp. 9280-9290. [cited by applicant]
Strehl et al., “An analysis of modelbased interval estimation for markov decision processes,” Journal of Computer and System Sciences, Dec. 2008, 74(8):1309-1331. [cited by applicant]
Sutton et al., “Policy gradient methods for reinforcement learning with function approximation,” Advances in Neural Information Processing Systems, Nov. 1999, 99:1057-1063. [cited by applicant]
Sutton, “Integrated architectures for learning, planning, and reacting based on approximating dynamic programming,” Proceedings of the Seventh International Conference on Machine Learning, Jun. 1990, pp. 216-224. [cited by applicant]
Thompson, “On the likelihood that one unknown probability exceeds another in view of the evidence of two samples,” Biometrika, Dec. 1933, 25(3/4):285-294. [cited by applicant]
Veeriah et al., “Discovery of useful questions as auxiliary tasks,” CoRR, Sep. 2019, arxiv.org/abs/1909.04607, 17 pages. [cited by applicant]
Wang et al., “Learning to reinforcement learn,” CoRR, Nov. 2016, arxiv.org/abs/1611.05763, 17 pages. [cited by applicant]
Watkins, “Learning from delayed rewards,” Thesis for the degree of Doctor, King's College, May 1989, 241 pages. [cited by applicant]
Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine Learning, May 1992, 8(3-4):229-256. [cited by applicant]
Wilson et al., “Humans use directed and random exploration to solve the explore-exploit dilemma,” Journal of Experimental Psychology: General, Dec. 2014, 143(6):2074-2081. [cited by applicant]
Xu et al., “Learning a prior over intent via meta-inverse reinforcement learning,” Proceedings of the 36th International Conference on Machine Learning, 2019, 97:6952-6962. [cited by applicant]
Xu et al., “Learning to explore via meta-policy gradient,” Proceedings of the 35th International Conference on Machine Learning, 2018, 80:5463-5472. [cited by applicant]
Xu et al., “Meta-gradient reinforcement learning,” CoRR, May 2018, arxiv.org/abs/1805.09801, 17 pages. [cited by applicant]
Zheng et al., “On learning intrinsic rewards for policy gradient methods,” CoRR, Jun. 2018, arxiv.org/abs/1804.06459, 15 pages. [cited by applicant]
Cited By (1)
US 12,705,477