IP Library › Granted Patent US 12,585,941
Granted Patent B2
US 12,585,941 · App. 17/570,870 · Granted Mar 24, 2026

Training a policy neural network for controlling an agent using best response policy iteration

Inventors: Thomas William Anthony (London, GB); Thomas Edward Eccles (London, GB); Andrea Tacchetti (London, GB); János Kramár (London, GB); Ian Michael Gemp (London, GB); Thomas Chalmers Hudson (London, GB); Nicolas Pierre Mickaël Porcel (London, GB); Marc Lanctot (London, GB); Julien Perolat (Saint Etienne, FR); Richard Everett (London, GB); Thore Kurt Hartwig Graepel (Cambridge, GB); Yoram Bachrach (London, GB)
Assignee: GDM Holding LLC
G06N3/08G06N3/045
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,585,941
App. No.
17/570,870
Granted
Mar 24, 2026
Kind
B2
Abstract

Methods, systems and apparatus, including computer programs encoded on computer storage media, for training a policy neural network by repeatedly updating the policy neural network at each of a plurality of training iterations. One of the methods includes generating training data for the training iteration by controlling the agent in accordance with an improved policy that selects actions in response to input state representations. A best response computation is performed using (i) a candidate policy generated from respective policy neural networks as of one or more preceding iterations and (ii) a candidate value neural network. The candidate value neural network is configured to generate a value output that is an estimate of a value of the environment being in the state characterized by a state representation to complete a particular task. The policy neural network is updated by training the policy neural network on the training data.

Claims (74)

1 . A method performed by one or more computers, the method comprising:

training a policy neural network by repeatedly updating the policy neural network at each of a plurality of training iterations, the policy neural network having a plurality of policy network parameters and configured to receive as input a state representation characterizing a state of an environment and to process the state representation in accordance with the policy network parameters to generate a policy output that defines a probability distribution over a set of actions that can be performed by an agent interacting with the environment, the training comprising, for each of the plurality of training iterations:

generating training data for the training iteration by controlling the agent in accordance with an improved policy that selects actions in response to input state representations by performing a best response computation, comprising:

determining a plurality of candidate actions using a candidate policy generated from respective policy neural networks as of one or more preceding iterations;

determining a value estimate for each candidate action using a candidate value neural network, wherein the candidate value neural network is configured to receive as input a state representation and the candidate action, and to process the state representation and the candidate action to generate a value output that is an estimate of a value of the environment being in the state characterized by the state representation and the agent performing the candidate action to complete a particular task; and

selecting, as the action to be performed by the agent in response to the state representation, the candidate action with a highest value estimate; and

updating the policy neural network by training the policy neural network on the training data.

2 . The method of claim 1 , wherein training the policy neural network further comprises repeatedly updating a value neural network at each of the plurality of training iterations, wherein the candidate value neural network is generated from respective value neural networks as of each of the one or more preceding iterations, and wherein the method further comprises:

updating the value neural network by training the value neural network on the training data.

3 . The method of claim 1 , wherein the environment comprises one or more other agents also interacting with the environment, and wherein performing the best response computation to select the action to be performed by the agent in response to the state representation comprises:

generating, using a base policy, a plurality of base profiles, each base profile comprising a respective action to be performed by each of the other agents in response to the state representation;

determining the plurality of candidate actions using the candidate policy, comprising:

processing the state representation using the candidate policy to generate a policy output; and

sampling the plurality of candidate actions from the probability distribution defined by the policy output;

determining the value estimate for each candidate action using the candidate value neural network, comprising:

for each base profile and for each candidate action:

determining a next state representation characterizing a predicted next state of the environment if the agent performs the candidate action and the other agents performed the actions in the base profile; and

processing the next state representation using the candidate value neural network to generate a value estimate for the base profile for the candidate action; and

determining the value estimate for each candidate action, wherein the value estimate for each candidate action comprises an overall value estimate from the value estimates for the base profiles for the candidate action; and

selecting, as the action to be performed by the agent in response to the state representation, the candidate action with the highest value estimate, comprising: selecting, as the action to be performed by the agent in response to the state representation, the candidate action with a highest overall value estimate.

4 . The method of claim 3 , wherein the base policy comprises a respective base policy for each other agent, and the respective base policy for each other agent is generated from respective policy neural networks as of one or more preceding iterations.

5 . The method of claim 3 , wherein determining the next state representation comprises determining the next state representation by using a simulator of the environment that processes the candidate action performed by the agent and the actions in the base profile performed by the other agents to generate the next state representation.

6 . The method of claim 1 , wherein updating the policy neural network by training the policy neural network on the training data comprises:

training the policy neural network using a supervised learning algorithm.

7 . The method of claim 1 , wherein the candidate policy is uniformly sampled among respective policy neural networks as of one or more preceding iterations.

8 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

training a policy neural network by repeatedly updating the policy neural network at each of a plurality of training iterations, the policy neural network having a plurality of policy network parameters and configured to receive as input a state representation characterizing a state of an environment and to process the state representation in accordance with the policy network parameters to generate a policy output that defines a probability distribution over a set of actions that can be performed by an agent interacting with the environment, comprising, for each of the plurality of training iterations:

generating training data for the training iteration by controlling the agent in accordance with an improved policy that selects actions in response to input state representations by performing a best response computation, comprising:

determining a plurality of candidate actions using a candidate policy generated from respective policy neural networks as of one or more preceding iterations;

determining a value estimate for each candidate action using a candidate value neural network, wherein the candidate value neural network is configured to receive as input a state representation and the candidate action, and to process the state representation and the candidate action to generate a value output that is an estimate of a value of the environment being in the state characterized by the state representation and the agent performing the candidate action to complete a particular task; and

selecting, as the action to be performed by the agent in response to the state representation, the candidate action with a highest value estimate; and

updating the policy neural network by training the policy neural network on the training data.

9 . The system of claim 8 , wherein training the policy neural network further comprises repeatedly updating a value neural network at each of the plurality of training iterations, wherein the candidate value neural network is generated from respective value neural networks as of each of the one or more preceding iterations, and wherein the operations further comprise:

updating the value neural network by training the value neural network on the training data.

10 . The system of claim 8 , wherein the environment comprises one or more other agents also interacting with the environment, and wherein performing the best response computation to select the action to be performed by the agent in response to the state representation comprises:

generating, using a base policy, a plurality of base profiles, each base profile comprising a respective action to be performed by each of the other agents in response to the state representation;

determining the plurality of candidate actions using the candidate policy, comprising:

processing the state representation using the candidate policy to generate a policy output; and

sampling the plurality of candidate actions from the probability distribution defined by the policy output;

determining the value estimate for each candidate action using the candidate value neural network, comprising:

for each base profile and for each candidate action:

determining a next state representation characterizing a predicted next state of the environment if the agent performs the candidate action and the other agents performed the actions in the base profile; and

processing the next state representation using the candidate value neural network to generate a value estimate for the base profile for the candidate action; and

determining the value estimate for each candidate action, wherein the value estimate for each candidate action comprises an overall value estimate from the value estimates for the base profiles for the candidate action; and

selecting, as the action to be performed by the agent in response to the state representation, the candidate action with the highest value estimate, comprising: selecting, as the action to be performed by the agent in response to the state representation, the candidate action with a highest overall value estimate.

11 . The system of claim 10 , wherein the base policy comprises a respective base policy for each other agent, and the respective base policy for each other agent is generated from respective policy neural networks as of one or more preceding iterations.

12 . The system of claim 10 , wherein determining the next state representation comprises determining the next state representation by using a simulator of the environment that processes the candidate action performed by the agent and the actions in the base profile performed by the other agents to generate the next state representation.

13 . The system of claim 8 , wherein updating the policy neural network by training the policy neural network on the training data comprises:

training the policy neural network using a supervised learning algorithm.

14 . The system of claim 8 , wherein the candidate policy is uniformly sampled among respective policy neural networks as of one or more preceding iterations.

15 . One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

training a policy neural network by repeatedly updating the policy neural network at each of a plurality of training iterations, the policy neural network having a plurality of policy network parameters and configured to receive as input a state representation characterizing a state of an environment and to process the state representation in accordance with the policy network parameters to generate a policy output that defines a probability distribution over a set of actions that can be performed by an agent interacting with the environment, comprising, for each of the plurality of training iterations:

generating training data for the training iteration by controlling the agent in accordance with an improved policy that selects actions in response to input state representations by performing a best response computation, comprising:

determining a plurality of candidate actions using a candidate policy generated from respective policy neural networks as of one or more preceding iterations;

determining a value estimate for each candidate action using a candidate value neural network, wherein the candidate value neural network is configured to receive as input a state representation and the candidate action, and to process the state representation and the candidate action to generate a value output that is an estimate of a value of the environment being in the state characterized by the state representation and the agent performing the candidate action to complete a particular task; and

selecting, as the action to be performed by the agent in response to the state representation, the candidate action with a highest value estimate; and

updating the policy neural network by training the policy neural network on the training data.

16 . The computer-readable storage media of claim 15 , wherein training the policy neural network further comprises repeatedly updating a value neural network at each of the plurality of training iterations, wherein the candidate value neural network is generated from respective value neural networks as of each of the one or more preceding iterations, and wherein the operations further comprise:

updating the value neural network by training the value neural network on the training data.

17 . The computer-readable storage media of claim 15 , wherein the environment comprises one or more other agents also interacting with the environment, and wherein performing the best response computation to select the action to be performed by the agent in response to the state representation comprises:

generating, using a base policy, a plurality of base profiles, each base profile comprising a respective action to be performed by each of the other agents in response to the state representation;

determining the plurality of candidate actions using the candidate policy, comprising:

processing the state representation using the candidate policy to generate a policy output; and

sampling the plurality of candidate actions from the probability distribution defined by the policy output;

determining the value estimate for each candidate action using the candidate value neural network, comprising:

for each base profile and for each candidate action:

determining a next state representation characterizing a predicted next state of the environment if the agent performs the candidate action and the other agents performed the actions in the base profile; and

processing the next state representation using the candidate value neural network to generate a value estimate for the base profile for the candidate action; and

determining the value estimate for each candidate action, wherein the value estimate for each candidate action comprises an overall value estimate from the value estimates for the base profiles for the candidate action; and

selecting, as the action to be performed by the agent in response to the state representation, the candidate action with the highest value estimate, comprising: selecting, as the action to be performed by the agent in response to the state representation, the candidate action with a highest overall value estimate.

18 . The computer-readable storage media of claim 17 , wherein the base policy comprises a respective base policy for each other agent, and the respective base policy for each other agent is generated from respective policy neural networks as of one or more preceding iterations.

19 . The computer-readable storage media of claim 17 , wherein determining the next state representation comprises determining the next state representation by using a simulator of the environment that processes the candidate action performed by the agent and the actions in the base profile performed by the other agents to generate the next state representation.

20 . The computer-readable storage media of claim 15 , wherein updating the policy neural network by training the policy neural network on the training data comprises:

training the policy neural network using a supervised learning algorithm.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 6, 2025
From: DEEPMIND TECHNOLOGIES LIMITED
To: GDM HOLDING LLC
Reel/Frame 071498/0210 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 12, 2022
From: ANTHONY, THOMAS WILLIAM; ECCLES, THOMAS EDWARD; TACCHETTI, ANDREA; KRAMAR, JANOS; GEMP, IAN MICHAEL; HUDSON, THOMAS CHALMERS; PORCEL, NICOLAS PIERRE MICKAEL; LANCTOT, MARC; PEROLAT, JULIEN; EVERETT, RICHARD; GRAEPEL, THORE KURT HARTWIG; BACHRACH, YORAM
To: DEEPMIND TECHNOLOGIES LIMITED
Reel/Frame 060790/0979 →
Continuity (2)
Provisional Application 63149032 · Feb 12, 2021
Related Publication 20220261635A1 · Aug 18, 2022
References Cited (124)
US 11580378B2 · Harmer · 2023 [cited by examiner]
CN 113421345A · 2021 [cited by examiner]
CN 113570040B · 2024 [cited by examiner]
Rajeswaran, Aravind, et al. “Epopt: Learning robust neural network policies using model ensembles.” arXiv preprint arXiv: 1610.01283 (2016). (Year: 2016). [cited by examiner]
Agmon et al., “Multi-robot perimeter patrol in adversarial settings,” 2008 IEEE International Conference on Robotics and Automation, May 2008, pp. 2339-2345. [cited by applicant]
Alchian et al., “Activity Analysis of Production and Allocation,” Cowles Commission for Research in Economics, 1951, 13(1):374-376. [cited by applicant]
Anthony et al., “Thinking fast and slow with deep learning and tree search.” Advances in Neural Information Processing Systems 30, 2017, 11 pages. [cited by applicant]
Arad et al., “Multi-dimensional iterative reasoning in action: The case of the Colonel Blotto game,” Journal of Economic Behavior & Organization, Nov. 2012, 84(2):571-585. [cited by applicant]
Axelrod et al., “The evolution of cooperation,” Science, Mar. 1981, 211(4489): 1390-1396. [cited by applicant]
Baarslag et al., “The first automated negotiating agents competition (ANAC 2010),” New Trends in Agent-Based Complex Automated Negotiations, Nov. 2011, pp. 113-135. [cited by applicant]
Baker et al., “Emergent tool use from multi-agent autocurricula,” CoRR, Sep. 2019, arXiv:1909.07528, 28 pages. [cited by applicant]
Balduzzi et al., “Open-ended learning in symmetric zero-sum games,” Proceedings of the 36th International Conference on Machine Learning, 2019, Jun. 2019, 97:434-443. [cited by applicant]
Balduzzi et al., “Re-evaluating evaluation,” Advances in Neural Information Processing Systems, 2018, pp. 3268-3279. [cited by applicant]
Balduzzi et al., “Smooth markets: A basic mechanism for organizing gradient-based learners,” CoRR, Jan. 2020, arXiv:2001.04678, 18 pages. [cited by applicant]
Bansal et al., “Emergent complexity via multi-agent competition,” CoRR, Oct. 2017, arXiv:1710.03748, 12 pages. [cited by applicant]
Bard et al., “The Hanabi challenge: A new frontier for AI research,” Artificial Intelligence, Mar. 2020, 280:103216. [cited by applicant]
Battaglia et al., “Relational inductive biases, deep learning, and graph networks,” CoRR, Jun. 2018, arXiv:1806.01261, 40 pages. [cited by applicant]
Berliner et al., “Backgammon computer program beats world champion,” Artificial Intelligence, Sep. 1980, 14(2):205-220. [cited by applicant]
Berner et al., “Dota 2 with large scale deep reinforcement learning,” CoRR, Dec. 2019, arxiv.org/abs/1912.06680, 66 pages. [cited by applicant]
Borel, “The game theory of play and integral equations with skew symmetric kernels,” Econometrica, Jan. 1953, 21:97-100. [cited by applicant]
Brown et al., “Superhuman AI for heads-up no-limit poker: Libratus beats top professionals.” Science, Dec. 2017, 359(6374):418-424. [cited by applicant]
Brown et al., “Superbuman AI for multiplayer poker,” Science, Jul. 2019, 365(6456):885-890. [cited by applicant]
Campbell et al., “Deep Blue,” Artificial Intelligence, Jan. 2002, 134(1-2):57-83. [cited by applicant]
Carroll et al., “On the utility of learning about humans for human-AI coordination,” Advances in Neural Information Processing Systems, 2019, 12 pages. [cited by applicant]
Celli et al., “Learning to correlate in multi-player general-sum sequential games, ” Advances in Neural Information Processing Systems, 2019, 11 pages. [cited by applicant]
Chen et al., “3-NASH is PPAD-Complete,” Electronic Colloquium on Computational Complexity, 2005, 134:12 pages. [cited by applicant]
Crandall et al., “Cooperating with machines,” Nature Communications, Jan. 2018, 9(1):1-12. [cited by applicant]
Daide.org.uk [online], “DAIDE Diplomacy Artificial Intelligence Development Environment,” Apr. 2, 2006, retrieved on May 21, 2012, retrieved from URL<http://www.daide.org.uk/>, 1 pages. [cited by applicant]
Daskalakis et al., “A counter-example to Karlin's strong conjecture for fictitious play;” 2014 IEEE 55th Annual Symposium on Foundations of Computer Science, Oct. 2014, 10 pages. [cited by applicant]
Daskalakis et al., “Three-player games are hard,” Electronic Colloquium on Computational Complexity, Nov. 2005, 10 pages. [cited by applicant]
De Jonge et al., “D-Brane: a Diplomacy playing agent for automated negotiations research,” Applied Intelligence, Feb. 2017, 47(1):158-177. [cited by applicant]
Deepmind.com [Online] “Alphastar Mastering the real-time strategy game Starcraft II,” Jan. 24, 2019, retrieved on Aug. 11, 2023, retrieved from URL<https://www.deepmind.com/blog/alphastar-mastering-the-real-time-strateg… [cited by applicant]
Eccles et al., “The imitation game: Learned reciprocity in markov games,” Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, May 2019, pp. 1934-1936. [cited by applicant]
Espeholt et al., “IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures,” Proceedings of the 35th International Conference on Machine Learning, 2018, 80:1407-1416. [cited by applicant]
Fabregues et al., “A testbed for multiagent systems Technical Report IIIA-TR-2009-09.” IIIA: Institut d'Investigaci en Intelligncia Artificial, Oct. 2009, 18 pages. [cited by applicant]
Farina et al., “Coarse correlation in extensiveform games,” Proceedings of the AAAI Conference on Artificial Intelligence, Apr. 2020, 34(2):1934-1941. [cited by applicant]
Ferreira et al., “DipBlue: A Diplomacy agent with strategic and trust reasoning,” ICAART 2015-7th International Conference on Agents and Artificial Intelligence, 2015, pp. 54-65. [cited by applicant]
Ferrucci, “Introduction to This is Watson,” IBM Journal of Research and Development, May 2012, 56(3.4):15 pages. [cited by applicant]
Findler, “Studies in machine cognition using the game of Poker,” Communications of the ACM, Apr. 1977, 20(4):230-245. [cited by applicant]
Foerster et al., “Bayesian action decoder for deep multi-agent reinforcement learning,” Proceedings of the 36th International Conference on Machine Learning, 2019, 97:1942-195. [cited by applicant]
Foerster et al., “Learning with opponent-learning awareness,” CoRR, Sep. 2017, arxiv.org/abs/1709.04326, 14 pages. [cited by applicant]
Fudenberg et al., “Learning and equilibrium,” Annual Review of Economics, Sep. 2009, 1:385-419. [cited by applicant]
Fudenberg et al., “Learning mixed equilibria,” Games and Economic Behavior, Jul. 1993, 5(3):320-367. [cited by applicant]
Genesereth et al., “General game playing: Overview of the AAAI competition,” AI magazine, Jun. 2005, 26(2):62-62. [cited by applicant]
Genugten, “A weakened form of fictitious play in two-person zero-sum games,” International Game Theory Review, 2000, 2(04):307-328. [cited by applicant]
Gibson, “Regret minimization in non-zero-sum games with applications to building champion multiplayer computer poker agents,” CoRR, Apr. 2013, arXiv:1305.0034, 48 pages. [cited by applicant]
Gordon et al., “No-regret learning in convex games,” Proceedings of the 25th international conference on Machine learning, Jul. 2008, pp. 360-367. [cited by applicant]
Greenblatt et al., “The Greenblatt Chess Program,” Proceedings of the fall joint computer conference, Nov. 1967, pp. 801-810. [cited by applicant]
Guss et al., “The MineRL competition on sample efficient reinforcement learning using human priors,” CoRR, Apr. 2019, arXiv:1904.10079, 28 pages. [cited by applicant]
Hall et al., “Thoughts on programming a diplomat,” Heuristic Programming in Artificial Intelligence. Nov. 1995, 3(9):123-145. [cited by applicant]
Harris, “On the rate of convergence of continuous-time fictitious play,” Games and Economic Behavior, Feb. 1998, 22(2):238-259. [cited by applicant]
Heinrich et al., “Deep reinforcement learning from self-play in imperfectinformation games.” CoRR, Mar. 2016, arXiv:1603.01121, 10 pages. [cited by applicant]
Heinrich et al., “Fictitious self-play in extensive-form games,” Proceedings of the 32nd International Conference on Machine Learning, 2015, pp. 37:805-813. [cited by applicant]
Herbrich et al., “TrueskillTM; a Bayesian skill rating system,” Advances in neural information processing systems 19, 2006, pp. 569-576. [cited by applicant]
Hochreiter et al., “Long short-term memory,” Neural computation, Nov. 1997, 9(8):1735-1780. [cited by applicant]
Hofbaner et al., “Evolutionary game dynamics,” Bulletin of the American Mathematical Society, 2003, 40(4):479-519. [cited by applicant]
Hofbauer et al., “On the global convergence of stochastic fictitious play,” Econometrica, Nov. 2002, 70(6):2265-2294. [cited by applicant]
Hu et al., “Simplified action decoder for deep multi-agent reinforcement learning,” CORR, Dec. 2019, arxiv.org/abs/1912.02288, 14 pages. [cited by applicant]
Hughes et al., “Learning to resolve alliance dilemmas in many-player zero-sum games,” CoRR, Feb. 2020, arXiv:2003.00799, 10 pages. [cited by applicant]
Jaderberg et al., “Human-level performance in 3D multiplayer games with population-based reinforcement learning.,” Science, May 2019, 364(6443):859-865. [cited by applicant]
Johansson et al., “Tactical coordination in no-press diplomacy;” Proceedings of the Fourth International Joint Conference on Autonomous Agents and Multiagent Systems, Jul. 2005, pp. 423-430. [cited by applicant]
Jonge et al., “The challenge of negotiation in the game of Diplomacy,” International Conference on Agreement Technologies, Apr. 2019, pp. 100-114. [cited by applicant]
Jordan, “Three problems in learning mixed-strategy Nash equilibria,” Games and Economic Behavior, Jul. 1993, 5(3):368-386. [cited by applicant]
Kempka et al., “VizDoom: A Doom-based AI research platform for visual reinforcement learning;” Conference on Computational Intelligence and Games, Sep. 2016, pp. 1-8. [cited by applicant]
Kipf et al., “Semi-supervised classification with graph convolutional networks,” CoRR, Sep. 2016, arXiv:1609.02907, 14 pages. [cited by applicant]
Knuth et al., “An analysis of Alpha-Beta pruning, ” Artificial Intelligence, 1975, 6(4):293-326. [cited by applicant]
Kohli et al., “Colonel Blotto on Facebook: The effect of social relations on strategic interaction.” WebSci'12: Proceedings of the 4th Annual ACM Web Science Conference, Jun. 2012, pp. 141-150. [cited by applicant]
Koutník et al., “Evolving large-scale neural networks for vision-based reinforcement learning,” Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, Jul. 2013, pp. 1061-1068. [cited by applicant]
Kraus et al., “Designing and building a negotiating automated agent,” Computational Intelligence, Feb. 1995, 11(1):132-171. [cited by applicant]
Kraus et al., “Diplomat, an agent in a multi agent environment; An overview,” IEEE International Performance Computing and Communications Conference, 1988, pp. 434-438. [cited by applicant]
Kraus et al., “Negotiation in a non-cooperative environment.” Journal of Experimental & Theoretical Artificial Intelligence, 1991, 3(4):255-281. [cited by applicant]
Kraus, “An automated Diplomacy player,” Heuristic Programming in Artificial Intelligence; The 1st Computer Olympiad, 1989, pp. 134-153. [cited by applicant]
Lanctot et al., “A unified game-theoretic approach to multiagent reinforcement learning.” Advances in Neural Information Processing Systems 30, 2017, pp. 4190-4203. [cited by applicant]
Lanctot et al., “OpenSpiel: A framework for reinforcement learning in games,” CoRR, Aug. 26, 2019, arxiv.org/abs/1908.09453, 27 pages. [cited by applicant]
Leibo et al., “Multiagent reinforcement learning in sequential social dilemmas,” CoRR, Feb. 10, 2017, arxiv.org/abs/1702.03037, 10 pages. [cited by applicant]
Lerer et al., “Improving policies via search in cooperative partially observable games,” CoRR, Dec. 5, 2019, arXiv:1912.02318, 15 pages. [cited by applicant]
Lerer et al., “Maintaining cooperation in complex social dilemmas using deep reinforcement learning.” CoRR, July 4. 2017, arXiv:1707.01068, 15 pages. [cited by applicant]
Leslie et al., “Generalised weakened fictitious play,” Games and Economic Behavior, Aug. 200656(2):285-298. [cited by applicant]
Lisy et al., “Equilibrium approximation quality of current no-limit Poker bots,” CoRR, Dec. 22, 2016, arxiv.org/abs/1612.07547, 6 pages. [cited by applicant]
Liu et al., “Emergent coordination through competition,” CoRR, Feb. 19, 2019, arxiv.org/abs/1902.07151, 19 pages. [cited by applicant]
Lockhart et al., “Computing approximate equilibria in sequential adversarial games by exploitability descent,” CoRR, Mar. 13, 2019, arXiv:1903.05614, 11 pages. [cited by applicant]
Lowe et al., “Multi-agent actor-critic for mixed cooperative-competitive environments,” Advances in Neural Information Processing Systems, 2017, pp. 6379-6390. [cited by applicant]
McKelvey et al., “Quantal response equilibria for normal form games,” Games and Economic Behavior, Jul. 1995, 10(1):6-38. [cited by applicant]
Mnih et al., “Asynchronous methods for deep reinforcement learning,” Proceedings of The 33rd International Conference on Machine Learning, 2016, 48:1928-1937. [cited by applicant]
Mnih et al., “Playing Atari with deep reinforcement learning,” CoRR, Dec. 19, 2013, arXiv:1312.5602, 9 pages. [cited by applicant]
Moravcík et al., “Deepstack: Expert-level artificial intelligence in heads-up no-limit Poker,” Science, Mar. 2, 2017, 356(6337):508-513. [cited by applicant]
Moulin et al., “Strategically zero-sum games: The class of games whose completely mixed equilibria cannot be improved upon,” International Journal of Game Theory, Sep. 1978, 7(3-4):201-221. [cited by applicant]
Nash et al., “Equilibrium points in n-person games,” Proceedings of the National Academy of Sciences, Jan. 1950, 36(1):48-49. [cited by applicant]
Ostrovski et al., “Payoff performance of fictitious play,” CoRR, Aug. 19, 2013, arXiv:1308.4049, 16 pages. [cited by applicant]
Papadimitrion, “On the complexity of the parity argument and other inefficient proofs of existence,” Journal of Computer and System Sciences, 48(3):498-532, 1994. [cited by applicant]
Paquette et al., “No-press Diplomacy: Modeling multi-agent gameplay,” Advances in Neural Information Processing Systems, 2019, pp. 4476-4487. [cited by applicant]
Park et al., “Deadlock avoidance in sequential resource allocation systems with multiple resource acquisitions and flexible routings,” IEEE Transactions on Automatic Control, Oct. 2001, 46(10):1572-1583. [cited by applicant]
Perolat et al. ,“From Poincaré recurrence to convergence in imperfect information games: Finding equilibrium via regularization,” Proceedings of the 38th International Conference on Machine Learning, 2020, 139:8525-8535. [cited by applicant]
Portugal et al., “A survey on multi-robot patrolling algorithms,” Doctoral Conference on Computing, Electrical and Industrial Systems, 2011, 349:139-146. [cited by applicant]
Resnick et al., “Pommerman: A multi-agent playground,” CoRR, Sep. 19, 2019, arXiv:1809.07124, 6 pages. [cited by applicant]
Reveliotis et al., “Polynomial-complexity deadlock avoidance policies for sequential resource allocation systems,” IEEE Transactions on Automatic Control, Oct. 1997, 42(10):1344-1357. [cited by applicant]
Roberson, “The Colonel Blotto game,” Economic Theory, Jan. 18, 2006, 29(1):1-24. [cited by applicant]
Robinson, “An iterative method of solving a game,” Annals of Mathematics, Sep. 1951, pp. 296-301. [cited by applicant]
Rooij et al., “Follow the leader if you can, hedge if you must,” Journal of Machine Learning Research, 2014, 15(1):1281-1316. [cited by applicant]
Rosenbloom, “A world-championship-level Othello program,” Artificial Intelligence, Nov. 1982, 19(3):279-320. [cited by applicant]
Ross et al., “A reduction of imitation learning and structured prediction to no-regret online learning,” Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, 2011, 15:627-635. [cited by applicant]
Samnel, “Some studies in machine learning using the game of Checkers,” IBM Journal of Research and Development, Jul. 1959, 3(3):210-229. [cited by applicant]
Schaeffer et al., “A world championship caliber Checkers program,” Artificial Intelligence, Feb. 1992, 53(2-3):273-289. [cited by applicant]
Scherrer et al., “Approximate modified policy iteration and its application to the game of Tetris,” Journal of Machine Learning Research, 2015, 16(49):1629-1676. [cited by applicant]
Schrandolph et al., “Temporal difference learning of position evahiation in the game of Go,” Advances in Neural Information Processing Systems 6 (NIPS 1993), 1994, pp. 817-824. [cited by applicant]
Serrino et al., “Finding friend and foe in multi-agent games,” Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 2019, pp. 1249-1259. [cited by applicant]
Shapiro et al., “Learning a game strategy using pattern-weights and self-play,” International Conference on Computers and Games, 2003, pp. 42-60. [cited by applicant]
Shapiro, “Note on a computation method in the theory of games,” Communications on Pure and Applied Mathematics, Nov. 1958, XI:587-593. [cited by applicant]
Silver et al., “A general reinforcement learning algorithm that masters Chess, Shogi, and Go through self-play,” Science, Dec. 7, 2018, 632(6419):1140-1144. [cited by applicant]
Silver et al., “Mastering the game of Go with deep neural networks and tree search,” Nature, Jan. 28, 2016, 529(7587):484. [cited by applicant]
Silver et al., “Mastering the game of Go without human knowledge,” Nature, Oct. 2017, 550(7676):354-359. [cited by applicant]
Sites.google.com [online], “Diplomacy AI—Albert,” Nov. 3, 2010, retrieved on Aug. 11, 2023, retrieved from URL<https://sites.google.com/site/diplomacyai>, 2 pages. [cited by applicant]
Smith, “Evolution and the Theory of Games,” American Scientist, Jan. 1976, 1:41-45. [cited by applicant]
Song et al., “Arena: A general evaluation platform and building toolkit for multi-agent intelligence,” Proceedings of the AAAI Conference on Artificial Intelligence, Apr. 3, 2020, 34(5):7253-7260. [cited by applicant]
Sutton et al., “Reinforcement Learning: An Introduction,” MIT Press, Nov. 13, 2018, 10 pages. [cited by applicant]
Taylor et al., “Evolutionary stable strategies and game dynamics,” Mathematical Biosciences, Jul. 1978, 40(1-2):145-156. [cited by applicant]
Tesauro, “TD-Gammon, a self-teaching Backgammon program, achieves master-level play,” Neural Computation, Mar. 1, 1994, 6(2):215-219. [cited by applicant]
Vandael et al., “Reinforcement learning of heuristic EV fleet charging in a day-ahead electricity market,” IEEE Transactions on Smart Grid, Jul. 4, 2015, 6(4):1795-1805. [cited by applicant]
Vinyals et al., “Grandmaster level in StarCraft II using multi-agent reinforcement learning,” Nature, Nov. 2019, 575(7782):350-354. [cited by applicant]
Waller et al., “Confidence intervals for weighted proportions,” Statistics in Medicine, May 30, 1994, 13(10):1071-1082. [cited by applicant]
Wang et al., “POET: open-ended coevolution of environments and their optimized solutions,” Proceedings of the Genetic and Evolutionary Computation Conference, Jul. 2019, pp. 142-151. [cited by applicant]
Web.inter.nl.net [online], “Diplomacy Adjudicator Test Cases,” Jan. 2002, retrieved on Apr. 21, 2023, retrieved from URL<http://web.inter.nl.net/users/L.B.Kruijswijk/>, 119 pages. [cited by applicant]
Webdiplomacy.net [online], “Diplomacy Adjudicator Test Cases,” Sep. 2009, retrieved on Apr. 21, 2023, retrieved from URL=https://webdiplomacy.net/datc.php>, 37 pages. [cited by applicant]
Wilson, “Probable inference, the law of succession, and statistical inference,” Journal of the American Statistical Association, May 8, 2012, 22(158):209-212. [cited by applicant]