IP Library › Granted Patent US 12,277,497
Granted Patent B2
US 12,277,497 · App. 18/131,753 · Granted Apr 15, 2025

Reinforcement learning using distributed prioritized replay

Inventors: David Budden (London, GB); Gabriel Barth-Maron (London, GB); John Quan (London, GB); Daniel George Horgan (London, GB)
Assignee: DeepMind Technologies Limited
G06N3/08G06N3/04G06N3/088G06N20/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,277,497
App. No.
18/131,753
Granted
Apr 15, 2025
Kind
B2
Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an action selection neural network used to select actions to be performed by an agent interacting with an environment. One of the systems includes (i) a plurality of actor computing units, in which each of the actor computing units is configured to maintain a respective replica of the action selection neural network and to perform a plurality of actor operations, and (ii) one or more learner computing units, in which each of the one or more learner computing units is configured to perform a plurality of learner operations.

Claims (66)

1. A system for training an action selection neural network having a plurality of network parameters and used to select actions to be performed by an agent interacting with an environment, the system being implemented using one or more computers and comprising:

a plurality of actor units, each of the actor units configured to maintain a respective replica of the action selection neural network and to perform actor operations in parallel with other actor units, the actor operations comprising:

receiving an observation characterizing a current state of an instance of the environment,

selecting an action to be performed by the agent using the action selection neural network replica and in accordance with current values of the network parameters,

obtaining transition data characterizing the environment instance subsequent to the agent performing the selected action,

generating a new experience tuple from the observation, the selected action, and the transition data,

determining an initial priority for the new experience tuple, comprising:

determining a learning error for the new experience tuple according to a reinforcement learning technique, and determining the initial priority from the learning error; and

storing the new experience tuple and the initial priority that is determined for the new experience tuple based on the learning error.

2. The system of claim 1 , wherein the new experience tuple and the initial priority are stored in a shared memory.

3. The system of claim 2 , further comprising one or more learner computing units, wherein each of the one or more learner computing units is configured to perform learner operations comprising:

sampling a batch of experience tuples from the shared memory based on the priorities for the experience tuples in the shared memory; and

determining, using the sampled experience tuples, an update to the network parameters using the reinforcement learning technique.

4. The system of claim 3 , wherein the learner operations further comprise:

determining for each sampled experience tuple a respective updated priority; and

updating the shared memory to associate the updated priorities with the sampled experience tuples.

5. The system of claim 3 , wherein the learner operations further comprise:

determining whether criteria for removing any experience tuples from the shared memory are satisfied; and

when the criteria are satisfied, updating the shared memory to remove one or more of the tuples.

6. The system of claim 3 , wherein the learner operations further comprise:

determining whether criteria for updating the actor units are satisfied; and

when the criteria are satisfied, transmitting updated parameter values to the actor units.

7. The system of claim 1 , wherein the initial priority is an absolute value of the learning error.

8. The system of claim 1 , wherein two or more of the actor units select actions using different exploration policies.

9. The system of claim 8 , wherein the different exploration policies are epsilon-greedy policies with different values of epsilon.

10. The system of claim 1 , wherein the reinforcement learning technique is an n-step Q learning technique or an actor-critic technique.

11. The system of claim 1 , wherein obtaining transition data characterizing the environment instance subsequent to the agent performing the selected action comprises:

selecting additional actions to be performed by the agent in response to subsequent observations using the action selection neural network replica to generate an n-step transition.

12. One or more non-transitory computer readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training an action selection neural network having a plurality of network parameters and used to select actions to be performed by an agent interacting with an environment, the operations comprising:

maintaining a plurality of actor units, each of the actor units configured to maintain a respective replica of the action selection neural network and to perform actor operations in parallel with other actor units; and

for each of the plurality of actor units, performing actor operations using the actor unit, the actor operations comprising:

receiving an observation characterizing a current state of an instance of the environment,

selecting an action to be performed by the agent using the action selection neural network replica and in accordance with current values of the network parameters,

obtaining transition data characterizing the environment instance subsequent to the agent performing the selected action,

generating a new experience tuple from the observation, the selected action, and the transition data,

determining an initial priority for the new experience tuple, comprising:

determining a learning error for the new experience tuple according to a reinforcement learning technique, and determining the initial priority from the learning error; and

storing the new experience tuple and the initial priority that is determined for the new experience tuple based on the learning error.

13. A computer-implemented method for training an action selection neural network having a plurality of network parameters and used to select actions to be performed by an agent interacting with an environment, the method comprising:

maintaining a plurality of actor units, each of the actor units configured to maintain a respective replica of the action selection neural network and to perform actor operations in parallel with other actor units; and

for each of the plurality of actor units, performing actor operations using the actor unit, the actor operations comprising:

receiving an observation characterizing a current state of an instance of the environment,

selecting an action to be performed by the agent using the action selection neural network replica and in accordance with current values of the network parameters,

obtaining transition data characterizing the environment instance subsequent to the agent performing the selected action,

generating a new experience tuple from the observation, the selected action, and the transition data,

determining an initial priority for the new experience tuple, comprising:

determining a learning error for the new experience tuple according to a reinforcement learning technique, and determining the initial priority from the learning error; and

storing the new experience tuple and the initial priority that is determined for the new experience tuple based on the learning error.

14. The method of claim 13 , wherein the new experience tuple and the initial priority are stored in a shared memory.

15. The method of claim 14 , further comprising:

maintaining one or more learner computing units; and

for each of the one or more learner computing units:

sampling, using the learner computing unit, a batch of experience tuples from the shared memory based on the priorities for the experience tuples in the shared memory; and

determining, using the sampled experience tuples, an update to the network parameters using the reinforcement learning technique.

16. The method of claim 15 , wherein for each of the one or more learner computing units, the method further comprises:

determining for each sampled experience tuple a respective updated priority; and

updating, using the learner computing unit, the shared memory to associate the updated priorities with the sampled experience tuples.

17. The method of claim 15 , wherein for each of the one or more learner computing units, the method further comprises:

determining whether criteria for removing any experience tuples from the shared memory are satisfied; and

when the criteria are satisfied, updating, using the learner computing unit, the shared memory to remove one or more of the tuples.

18. The method of claim 15 , wherein the reinforcement learning technique is an n-step Q learning technique or an actor-critic technique.

19. The method of claim 15 , wherein for each of the one or more learner computing units, the method further comprises:

determining whether criteria for updating the actor units are satisfied; and

when the criteria are satisfied, transmitting updated parameter values to the actor units.

20. The method of claim 15 , wherein obtaining transition data characterizing the environment instance subsequent to the agent performing the selected action comprises:

selecting additional actions to be performed by the agent in response to subsequent observations using the action selection neural network replica to generate an n-step transition.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 6, 2025
From: DEEPMIND TECHNOLOGIES LIMITED
To: GDM HOLDING LLC
Reel/Frame 071498/0210 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 21, 2023
From: BUDDEN, DAVID; BARTH-MARON, GABRIEL; QUAN, JOHN; HORGAN, DANIEL GEORGE
To: DEEPMIND TECHNOLOGIES LIMITED
Reel/Frame 064990/0573 →
Continuity (3)
Continuation 16641751
Provisional Application 62578384 · Oct 27, 2017
Related Publication 20230252288A1 · Aug 10, 2023
References Cited (55)
US 11625604B2 · Budden · 2023 [cited by examiner]
US 20160232445A1 · Srinivasan et al. · 2016 [cited by applicant]
US 20170286860A1 · Chen · 2017 [cited by examiner]
US 20200234117A1 · Hafner · 2020 [cited by examiner]
CN 103248693 · 2013 [cited by applicant]
CN 105637540 · 2016 [cited by applicant]
CN 106056213 · 2016 [cited by applicant]
CN 107209872 · 2017 [cited by applicant]
Abadi et al., “TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems,” https://arxiv.org/abs/1603.04467, Mar. 2016, 19 pages. [cited by applicant]
Alain et al., “Variance Reduction in SGD by Distributed Importance Sampling,” https://arxiv.org/abs/1511.06481v1, Nov. 2015, 14 pages. [cited by applicant]
Babaeizadeh et al., “Reinforcement Learning through Asynchronous Advantage Actor-Critic on a GPU,” https://arxiv.org/abs/1611.06256v3, last revised Mar. 2017, 12 pages. [cited by applicant]
Barth-Maron et al., “Distributed Distributional Deterministic Policy Gradients,” https://arxiv.org/abs/1804.08617, Apr. 2018, 16 pages. [cited by applicant]
Bellemare et al., “A Distributional Perspective on Reinforcement Learning,” https://arxiv.org/abs/1707.06887, Jul. 2017, 19 pages. [cited by applicant]
Bellemare et al., “The Arcade Learning Environment: An Evaluation Platform for General Agents,” J. Artif. Intell. Res. (JAIR), Jun. 2013, 47:253-279. [cited by applicant]
Clemente et al., “Efficient Parallel Methods for Deep Reinforcement Learning,” https://arxiv.org/abs/1705.04862, May 2017, 9 pages. [cited by applicant]
Dean et al., “Large Scale Distributed Deep Networks,” Proceedings of the 25th International Conference on Neural Information Processing Systems, Dec. 2012, 9 pages. [cited by applicant]
Deng et al., “ImageNet: A Large-Scale Hierarchical Image Database,” 2009 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2009, 248-255. [cited by applicant]
Gruslys et al., “The Reactor: A Sample-Efficient Actor-Critic Architecture,” https://arxiv.org/abs/1704.04651v1, Apr. 2017, 12 pages. [cited by applicant]
Hastings, “Monte Carlo Sampling Methods Using Markov Chains and Their Applications,” Biometrika, Apr. 1970, 57(1):97-109. [cited by applicant]
Heess et al., “Emergence of Locomotion Behaviours in Rich Environments,” https://arxiv.org/abs/1707.02286, Jul. 2017, 14 pages. [cited by applicant]
Hessel et al., “Rainbow: Combining Improvements in Deep Reinforcement Learning,” https://arxiv.org/abs/1710.02298, Oct. 2017, 14 pages. [cited by applicant]
Hester et al., “Learning from Demonstrations for Real World Reinforcement Learning,” https://arxiv.org/abs/1704.03732v1, Apr. 2017, 11 pages. [cited by applicant]
Hinton, “To recognize shapes, first learn to generate images,” Progress in Brain Research, 2007, 165:535-547. [cited by applicant]
Horgan et al., “Distributed Prioritized Experience Replay,” https://arxiv.org/abs/1803.00933, Mar. 2018, 19 pages. [cited by applicant]
Jaderberg et al., “Reinforcement Learning with Unsupervised Auxiliary Tasks,” https://arxiv.org/abs/1611.05397, Nov. 2016, 14 pages. [cited by applicant]
Kaiser et al., “One Model To Learn Them All,” https://arxiv.org/abs/1706.05137, Jun. 2017, 10 pages. [cited by applicant]
Kingma et al., “Adam: A Method for Stochastic Optimization,” https://arxiv.org/abs/1412.6980v1, Dec. 2014, 9 pages. [cited by applicant]
Krizhevsky, “One weird trick for parallelizing convolutional neural networks,” https://arxiv.org/abs/1404.5997, Apr. 2014, 7 pages. [cited by applicant]
Lillicrap et al., “Continuous control with deep reinforcement learning,” https://arxiv.org/abs/1509.02971v5, Feb. 2016, 14 pages. [cited by applicant]
Lin, “Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Reaching,” Machine Learning, May 1992, 293-321. [cited by applicant]
Loshchilov et al., “Online Batch Selection for Faster Training of Neural Networks,” https://arxiv.org/abs/1511.06343v1, Nov. 2015, 12 pages. [cited by applicant]
Mnih et al., “Asynchronous Methods for Deep Reinforcement Learning,” https://arxiv.org/abs/1602.01783v1, Feb. 2016, 28 pages. [cited by applicant]
Mnih et al., “Asynchronous Methods for Deep Reinforcement Learning,” International Conference on Machine Learning, Jun. 2016, 10 pages. [cited by applicant]
Mnih et al., “Human-level control through deep reinforcement learning,” Nature, Feb. 2015, 518(7540):529-533. [cited by applicant]
Moore et al., “Prioritized Sweeping: Reinforcement Learning with Less Data and Less Time,” Machine Learning, Oct. 1993, 13:103-130. [cited by applicant]
Nair et al., “Massively Parallel Methods for Deep Reinforcement Learning,” https://arxiv.org/abs/1507.04296, Jul. 2015, 14 pages. [cited by applicant]
Office Action in Chinese Appln. No. 201880068801.8, dated Jan. 13, 2023, 28 pages (with English translation). [cited by applicant]
PCT International Preliminary Report on Patentability in International Appln. No. PCT/EP2018/079566, mailed May 7, 2020, 12 pages. [cited by applicant]
PCT International Search Report and Written Opinion in International Appln. No. PCT/EP2018/079566, mailed Feb. 4, 2019, 19 pages. [cited by applicant]
Prokhorov et al., “Adaptive Critic Designs,” IEEE Transactions on Neural Networks, Sep. 1997, 8(5):997-1007. [cited by applicant]
Riedmiller, “Neural Fitted Q Iteration—First Experiences with a Data Efficient Neural Reinforcement Learning Method,” European Conference on Machine Learning, Oct. 2005, 317-328. [cited by applicant]
Schaul et al., “Prioritized Experience Replay,” https://arxiv.org/abs/1511.05952v2, Nov. 2015, 23 pages. [cited by applicant]
Schaul et al., “Prioritized Experience Replay,” https://arxiv.org/abs/1511.05952v4, Feb. 2016, 21 pages. [cited by applicant]
Silver et al., “Concurrent Reinforcement Learning from Customer Interactions,” Proceedings of the 30th International Conference on Machine Learning, Jun. 2013, 9 pages. [cited by applicant]
Silver et al., “Deterministic Policy Gradient Algorithms,” Proceedings of the 31st International Conference on Machine Learning, Jun. 2014, 9 pages. [cited by applicant]
Silver et al., “Mastering the game of go with deep neural networks and tree search,” Nature, Jan. 2016, 529:484-489. [cited by applicant]
Sutton, “Learning to Predict by the Methods of Temporal Differences,” Machine Learning, Aug. 1988, 3:9-44. [cited by applicant]
Tassa et al., “DeepMind Control Suite,” https://arxiv.org/abs/1801.00690, Jan. 2018, 24 pages. [cited by applicant]
Todorov et al., “MuJoCo: A physics engine for model-based control,” 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Oct. 2012, 5026-5033. [cited by applicant]
Uhlenbeck et al., “On the Theory of the Brownian Motion,” Physical Review, Sep. 1930, 36(5):823-841. [cited by applicant]
Van Hasselt et al., “Deep Reinforcement Learning with Double Q-Learning,” Thirtieth AAAI Conference on Artificial Intelligence, Mar. 2016, 2094-2100. [cited by applicant]
Van Hasselt, “Double Q-learning,” Proceedings of the 23rd International Conference on Neural Information Processing Systems, Dec. 2010, 9 pages. [cited by applicant]
Wang et al., “Dueling Network Architectures for Deep Reinforcement Learning,” https://arxiv.org/abs/1511.06581v3, last revised Apr. 2016, 15 pages. [cited by applicant]
Wang et al., “Sample Efficient Actor-Critic with Experience Replay,” https://arxiv.org/abs/1611.01224v2, Jul. 2017, 20 pages. [cited by applicant]
Wiering et al., “Ensemble Algorithms in Reinforcement Learning,” IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), Aug. 2008, 38(4): 930-936. [cited by applicant]