IP Library › Granted Patent US 12,572,803
Granted Patent B2
US 12,572,803 · App. 18/771,770 · Granted Mar 10, 2026

Multi-agent reinforcement learning with matchmaking policies

Inventors: David Silver (Hitchin, GB); Oriol Vinyals (London, GB); Maxwell Elliot Jaderberg (London, GB)
Assignee: GDM Holding LLC
G06N3/08G06F18/214H04L63/205
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,572,803
App. No.
18/771,770
Granted
Mar 10, 2026
Kind
B2
Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a policy neural network having a plurality of policy parameters and used to select actions to be performed by an agent to control the agent to perform a particular task while interacting with one or more other agents in an environment. In one aspect, the method includes: maintaining data specifying a pool of candidate action selection policies; maintaining data specifying respective matchmaking policy; and training the policy neural network using a reinforcement learning technique to update the policy parameters. The policy parameters define policies to be used in controlling the agent to perform the particular task.

Claims (39)

1 . A computed-implemented method comprising:

maintaining a pool of candidate action selection policies;

maintaining, for each candidate action selection policy in the pool, a respective matchmaking policy that is configured to define a score distribution over the pool of candidate action selection policies, wherein the score distribution has a respective score for each of one or more previously selected candidate action selection policies of the candidate action selection policies that have been previously selected for generating training data, and wherein, for each of the one or more previously selected candidate action selection policies, the respective matchmaking policy is configured such that the respective score is dependent on a level of performance of the previously selected candidate action selection policy in controlling an agent to perform a task when interacting with one or more other agents that were controlled by other previously selected candidate action selection policies;

training one or more of the candidate action selection policies in the pool using one or more of the matchmaking policies, wherein the training comprises, for a particular candidate action selection policy:

selecting, in accordance with the score distribution defined by the respective matchmaking policy for the particular candidate action selection policy, one or more candidate action selection policies from the pool;

generating training data for the particular candidate action selection policy by causing an agent controlled using the particular candidate action selection policy to perform the task while interacting with one or more other agents that are controlled by the selected one or more candidate action selection policies; and

updating the particular candidate action selection policy through reinforcement learning by using the training data.

2 . The method of claim 1 , wherein the candidate action selection policies are each defined by a respective set of neural network parameters, and wherein updating the particular candidate action selection policy comprises updating values of a respective set of neural network parameters that define the particular candidate action selection policy.

3 . The method of claim 1 , wherein the respective score for each of the one or more previously selected candidate action selection policies comprises a probability score.

4 . The method of claim 1 , wherein the respective score for each of the one or more previously selected candidate action selection policies comprises a weighted probability score that is computed by multiplying a weight value with a probability score, wherein the weight value is computed using a weighting function.

5 . The method of claim 4 , wherein the weight values to be multiplied with different probability scores are different.

6 . The method of claim 4 , wherein the weight values to be multiplied with different probability scores are the same.

7 . The method of claim 3 , wherein for each of one or more previously selected candidate action selection policies, the respective score is configured to be proportional to the level of performance of the previously selected candidate action selection policy in controlling the agent to perform the task.

8 . The method of claim 4 , wherein for each of one or more previously selected candidate action selection policies, the weight value to be multiplied with the respective score is configured to be proportional to the performance of the previously selected candidate action selection policy in controlling the agent to perform the task.

9 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

maintaining a pool of candidate action selection policies;

maintaining, for each candidate action selection policy in the pool, a respective matchmaking policy that is configured to define a score distribution over the pool of candidate action selection policies, wherein the score distribution has a respective score for each of one or more previously selected candidate action selection policies of the candidate action selection policies that have been previously selected for generating training data, and wherein, for each of the one or more previously selected candidate action selection policies, the respective matchmaking policy is configured such that the respective score is dependent on a level of performance of the previously selected candidate action selection policy in controlling an agent to perform a task when interacting with one or more other agents that were controlled by other previously selected candidate action selection policies;

training one or more of the candidate action selection policies in the pool using one or more of the matchmaking policies, wherein the training comprises, for a particular candidate action selection policy:

selecting, in accordance with the score distribution defined by the respective matchmaking policy for the particular candidate action selection policy, one or more candidate action selection policies from the pool;

generating training data for the particular candidate action selection policy by causing an agent controlled using the particular candidate action selection policy to perform the task while interacting with one or more other agents that are controlled by the selected one or more candidate action selection policies; and

updating the particular candidate action selection policy through reinforcement learning by using the training data.

10 . The system of claim 9 , wherein the candidate action selection policies are each defined by a respective set of neural network parameters, and wherein updating the particular candidate action selection policy comprises updating values of a respective set of neural network parameters that define the particular candidate action selection policy.

11 . The system of claim 9 , wherein the respective score for each of the one or more previously selected candidate action selection policies comprises a probability score.

12 . The system of claim 9 , wherein the respective score for each of the one or more previously selected candidate action selection policies comprises a weighted probability score that is computed by multiplying a weight value with a probability score, wherein the weight value is computed using a weighting function.

13 . The system of claim 12 , wherein the weight values to be multiplied with different probability scores are different.

14 . The system of claim 12 , wherein the weight values to be multiplied with different probability scores are the same.

15 . The system of claim 11 , wherein for each of one or more previously selected candidate action selection policies, the respective score is configured to be proportional to the level of performance of the previously selected candidate action selection policy in controlling the agent to perform the task.

16 . The system of claim 12 , wherein for each of one or more previously selected candidate action selection policies, the weight value to be multiplied with the respective score is configured to be proportional to the performance of the previously selected candidate action selection policy in controlling the agent to perform the task.

17 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

maintaining a pool of candidate action selection policies;

maintaining, for each candidate action selection policy in the pool, a respective matchmaking policy that is configured to define a score distribution over the pool of candidate action selection policies, wherein the score distribution has a respective score for each of one or more previously selected candidate action selection policies of the candidate action selection policies that have been previously selected for generating training data, and wherein, for each of the one or more previously selected candidate action selection policies, the respective matchmaking policy is configured such that the respective score is dependent on a level of performance of the previously selected candidate action selection policy in controlling an agent to perform a task when interacting with one or more other agents that were controlled by other previously selected candidate action selection policies;

training one or more of the candidate action selection policies in the pool using one or more of the matchmaking policies, wherein the training comprises, for a particular candidate action selection policy:

selecting, in accordance with the score distribution defined by the respective matchmaking policy for the particular candidate action selection policy, one or more candidate action selection policies from the pool;

generating training data for the particular candidate action selection policy by causing an agent controlled using the particular candidate action selection policy to perform the task while interacting with one or more other agents that are controlled by the selected one or more candidate action selection policies; and

updating the particular candidate action selection policy through reinforcement learning by using the training data.

18 . The non-transitory computer storage media of claim 17 , wherein the candidate action selection policies are each defined by a respective set of neural network parameters, and wherein updating the particular candidate action selection policy comprises updating values of a respective set of neural network parameters that define the particular candidate action selection policy.

19 . The non-transitory computer storage media of claim 17 , wherein the respective score for each of the one or more previously selected candidate action selection policies comprises a probability score.

20 . The non-transitory computer storage media of claim 17 , wherein the respective score for each of the one or more previously selected candidate action selection policies comprises a weighted probability score that is computed by multiplying a weight value with a probability score, wherein the weight value is computed using a weighting function.

21 . The non-transitory computer storage media of claim 20 , wherein the weight values to be multiplied with different probability scores are different.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 6, 2025
From: DEEPMIND TECHNOLOGIES LIMITED
To: GDM HOLDING LLC
Reel/Frame 071498/0210 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 11, 2025
From: SILVER, DAVID; VINYALS, ORIOL; JADERBERG, MAXWELL ELLIOT
To: DEEPMIND TECHNOLOGIES LIMITED
Reel/Frame 070820/0495 →
Continuity (5)
Continuation 18131567 · Apr 6, 2023
Continuation 16752496 · Jan 24, 2020
Provisional Application 62894633 · Aug 30, 2019
Provisional Application 62796567 · Jan 24, 2019
Related Publication 20240370725A1 · Nov 7, 2024
References Cited (75)
US 11423300B1 · Ritter · 2022 [cited by examiner]
US 11627165B2 · Silver · 2023 [cited by examiner]
US 12067491B2 · Silver · 2024 [cited by examiner]
US 20180165603A1 · Van Seijen · 2018 [cited by applicant]
US 20190102676A1 · Nazari et al. · 2019 [cited by applicant]
US 20190244099A1 · Schaul · 2019 [cited by examiner]
US 20190266489A1 · Hu · 2019 [cited by applicant]
US 20190354867A1 · Czarnecki · 2019 [cited by examiner]
US 20200033868A1 · Palanisamy et al. · 2020 [cited by applicant]
US 20200104680A1 · Reed · 2020 [cited by examiner]
US 20200160168A1 · Yang · 2020 [cited by applicant]
WO WO2018224695 · 2018 [cited by applicant]
Human-level performance in first-person multiplayer games with population-based deep reinforcement learning Max Jaderberg et al. (Year: 2018). [cited by examiner]
Balduzzi et al., “Open-ended Learning in Symmetric Zero-sum Games,” Proceedings of the 36th International Conference on Machine Learning, May 2019, 10 pages. [cited by applicant]
Brown, “Iterative Solution of Games by Fictitious Play,” Activity Analysis of Production and Allocation, 1951, Chapter 24, 374-376. [cited by applicant]
Buro, “ORTS: A Hack-Free RTS Game Environment,” International Conference on Computers and Games, 2002, 280-291. [cited by applicant]
Buro, “Real-Time Strategy Games: A New AI Research Challenge,” Intl Joint Conf. Artificial Intelligence, Aug. 2003, 2 pages. [cited by applicant]
Campbell et al., “Deep Blue,” Artificial Intelligence, Jan. 2002, 134(1-2):57-83. [cited by applicant]
Christiano et al., “Deep Reinforcement Learning from Human Preferences,” Proceedings of the 31st International Conference on Neural Information Processing Systems, Dec. 2017, 9 pages. [cited by applicant]
Churchill et al., “An Analysis of Model-Based Heuristic Search Techniques for StarCraft Combat Scenarios,” Artificial Intelligence and Interactive Digital Entertainment Conf., Sep. 2017, 7 pages. [cited by applicant]
Czarnecki et al., “Mix&Match—Agent Curricula for Reinforcement Learning,” https://arxiv.org/abs/1806.01780, Jun. 2018, 12 pages. [cited by applicant]
deepmind.com [online], “AlphaStar: Mastering the Real-Time Strategy Game StarCraft II,” Jan. 24, 2019, retrieved on Apr. 21, 2020, retrieved from URL <https://deepmind.com/blog/article/alphastar-mastering-real-time-stra… [cited by applicant]
Espeholt et al., “IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures,” https://arxiv.org/abs/1802.01561v3, last revised Jun. 2018, 22 pages. [cited by applicant]
Farooq et al., “StarCraft AI Competition: A Step Toward Human-Level AI for Real-Time Strategy Games,” AI Magazine, Jul. 2016, 37:102-107. [cited by applicant]
He et al., “Deep Residual Learning for Image Recognition,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2016, 770-778. [cited by applicant]
Heinrich et al., “Fictitious Self-Play in Extensive-Form Games,” Proceedings of the 32nd International Conference on Machine Learning, Jun. 2015, 9 pages. [cited by applicant]
Hinton et al., “Distilling the Knowledge in a Neural Network,” https://arxiv.org/abs/1503.02531, Mar. 2015, 9 pages. [cited by applicant]
Hochreiter et al., “Long Short-Term Memory,” Neural Computation, Nov. 1997, 9(8):1735-1780. [cited by applicant]
Hsieh et al., “Building a Player Strategy Model by Analyzing Replays of Real-Time Strategy Games,” 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Jun. 20… [cited by applicant]
Ibarz et al., “Reward learning from human preferences and demonstrations in Atari,” Proceedings of the 32nd International Conference on Neural Information Processing Systems, Dec. 2018, 13 pages. [cited by applicant]
International Preliminary Report on Patentability in International Appln. No. PCT/EP2020/051839, dated Aug. 5, 2021, 12 pages. [cited by applicant]
Jaderberg et al., “Human-level performance in 3D multiplayer games with population-based reinforcement learning,” Science, May 2019, 364(6443):7 pages. [cited by applicant]
Jouppi et al., “In-Datacenter Performance Analysis of a Tensor Processing Unit,” https://arxiv.org/abs/1704.04760, Apr. 2017, 17 pages. [cited by applicant]
Justesen et al., “Learning Macromanagement in StarCraft from Replays using Deep Learning,” 2017 IEEE Conference on Computational Intelligence and Games (CIG), Aug. 2017, 162-169. [cited by applicant]
Kingma et al., “Adam: A Method for Stochastic Optimization,” https://arxiv.org/abs/1412.6980v9, last revised Jan. 2017, 15 pages. [cited by applicant]
Lanctot et al., “A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning,” Proceedings of the 31st International Conference on Neural Information Processing Systems, Dec. 2017, 14 pages. [cited by applicant]
Lecun et al., “Deep learning,” Nature 521, May 2015, 436-444. [cited by applicant]
Leslie et al., “Generalised weakened fictitious play,” Games and Economic Behavior, Aug. 2006, 56(2):285-298. [cited by applicant]
Metz et al., “Discrete Sequential Prediction of Continuous Actions for Deep RL,” https://arxiv.org/abs/1705.05035v1, May 2017, 24 pages. [cited by applicant]
Mikolov et al., “Recurrent Neural Network Based Language Model,” Interspeech 2010, Sep. 2010, 1045-1048. [cited by applicant]
Mnih et al., “Asynchronous Methods for Deep Reinforcement Learning,” https://arxiv.org/abs/1602.01783v2, last revised Jun. 2016, 19 pages. [cited by applicant]
Mnih et al., “Asynchronous Methods for Deep Reinforcement Learning,” Proceedings of The 33rd International Conference on Machine Learning, Jun. 2016, 10 pages. [cited by applicant]
Mnih et al., “Human-level control through deep reinforcement learning,” Nature, Feb. 2015, 518:529-533. [cited by applicant]
Nair et al., “Overcoming Exploration in Reinforcement Learning with Demonstrations,” 2018 IEEE International Conference on Robotics and Automation (ICRA), May 2018, 6292-6299. [cited by applicant]
Office Action in European Appln. No. 20702116.3, dated May 27, 2024, 9 pages. [cited by applicant]
Oh et al., “Self-Imitation Learning,” https://arxiv.org/abs/1806.05635, Jun. 2018, 13 pages. [cited by applicant]
Parisotto et al., “Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning,” https://arxiv.org/abs/1511.06342v4, last revised Feb. 2016, 16 pages. [cited by applicant]
Pathak et al., “Curiosity-Driven Exploration by Self-Supervised Prediction,” 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Jul. 2017, 488-489. [cited by applicant]
PCT International Search Report and Written Opinion in International Appln. No. PCT/EP2020/051839, mailed May 13, 2020, 18 pages. [cited by applicant]
Perez et al., “FiLM: Visual Reasoning with a General Conditioning Layer,” https://arxiv.org/abs/1709.07871v2, last revised Dec. 2017, 13 pages. [cited by applicant]
Pourchot et al., “CEM-RL: Combining evolutionary and gradient-based methods for policy search,” https://arxiv.org/abs/1810.01222v1, Oct. 2018, 17 pages. [cited by applicant]
Precup et al., “Eligibility Traces for Off-Policy Policy Evaluation,” ICML '00 Proc. 17th Intl Conf. Machine Learning, retrieved from URL <https://scholarworks.umass.edu/cgi/viewcontent.cgi?article=1079&context=cs facul… [cited by applicant]
Rusu et al., “Policy Distillation,” https://arxiv.org/abs/1511.06295v2, last revised Jan. 2016, 13 pages. [cited by applicant]
Samvelyan et al., “The StarCraft Multi-Agent Challenge,” Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, May 2019, 2186-2188. [cited by applicant]
Schulman et al., “Proximal Policy Optimization Algorithms,” https://arxiv.org/abs/1707.06347v2, last revised Aug. 2017, 12 pages. [cited by applicant]
Shao et al., “StarCraft Micromanagement With Reinforcement Learning and Curriculum Transfer Learning,” IEEE Transactions on Emerging Topics in Computational Intelligence, Feb. 2019, 12 pages. [cited by applicant]
Silver et al., “A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play,” Science, Dec. 2018, 362(6419):6 pages. [cited by applicant]
Silver et al., “Mastering the game of Go with deep neural networks and tree search,” Nature, Jan. 2016, 529:484-489. [cited by applicant]
starcraft2.com [online], “DeepMind Research on Ladder,” Jul. 2019, retrieved on May 7, 2020, retrieved from URL <https://starcraft2.com/en-us/news/22933138>, 6 pages. [cited by applicant]
Sun et al., “TStarBots: Defeating the Cheating Level Builtin AI in StarCraft II in the Full Game,” https://arxiv.org/abs/1809.07193v3, last revised Dec. 2018, 24 pages. [cited by applicant]
Sutton, “Learning to Predict by the Method of Temporal Differences,” Machine Learning, Aug. 1988, 3:9-44. [cited by applicant]
Synnaeve et al., “A Bayesian Model for Plan Recognition in RTS Games applied to StarCraft,” Artificial Intelligence and Interactive Digital Entertainment Conf., Nov. 2011, 7 pages. [cited by applicant]
Synnaeve et al., “Forward Modeling for Partial Observation Strategy Games—A StarCraft Defogger,” Advances in Neural Information Processing Systems 31 (NIPS 2018), Dec. 2018, 11 pages. [cited by applicant]
torchcraft.github.io [online], “TorchCraftAI,” Nov. 2018, retrieved on May 7, 2020, retrieved from URL <https://torchcraft.github.io/TorchCraftAI/>, 2 pages. [cited by applicant]
Uchibe, “Cooperative and Competitive Reinforcement and Imitation Learning for a Mixture of Heterogeneous Learning Modules,” Front. Neurorobot., Sep. 2018, 12(61):11 pages. [cited by applicant]
Uriarte et al., “Improving Monte Carlo Tree Search Policies in StarCraft via Probabilistic Models Learned from Replay Data,” Artificial Intelligence and Interactive Digital Entertainment Conf., Jan. 2016, 100-106. [cited by applicant]
Usunier et al., “Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks,” https://arxiv.org/abs/1609.02993v3, Nov. 2016, 18 pages. [cited by applicant]
Vaswani et al., “Attention is All you Need,” Advances in Neural Information Processing Systems 30 (NIPS 2017), Dec. 2017, 11 pages. [cited by applicant]
Vinyals et al., “AlphaStar: Mastering the Real-Time Strategy Game StarCraft II,” retrieved from URL<https://deepmind.com/blog/article/alphastar-mastering-real-time-strategy-game-starcraft-ii>, Jan. 2019, retrieved on Ma… [cited by applicant]
Vinyals et al., “Pointer Networks,” Adv. Neural Information Process. Syst. 28, Dec. 2015, 9 pages. [cited by applicant]
Vinyals et al., “StarCraft II: A New Challenge for Reinforcement Learning,” https://arxiv.org/abs/1708.04782, Aug. 2017, 20 pages. [cited by applicant]
Wang et al., “Sample Efficient Actor-Critic with Experience Replay,” Phttps://arxiv.org/abs/1611.01224v2, last revised Jul. 2017, 20 pages. [cited by applicant]
Weber et al., “Case-Based Reasoning for Build Order in Real-Time Strategy Games,” Proceedings of the Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Oct. 2009, 6 pages. [cited by applicant]
Wolski et al., “OpenAI Five,” retrieved from URL <https://blog.openai.com/openai-five/>, Jun. 2018, 10 pages. [cited by applicant]
Zambaldi et al., “Relational Deep Reinforcement Learning,” https://arxiv.org/abs/1806.01830v2, Jun. 2018, 15 pages. [cited by applicant]