IP Library › Granted Patent US 12,260,334
Granted Patent B2
US 12,260,334 · App. 18/497,924 · Granted Mar 25, 2025

Neural programming

Inventors: Scott Ellison Reed (Atlanta, GA); Joao Ferdinando Gomes de Freitas (London, GB)
Assignee: DeepMind Technologies Limited
G06N3/08G06N3/044G06N20/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,260,334
App. No.
18/497,924
Granted
Mar 25, 2025
Kind
B2
Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for neural programming. One of the methods includes processing a current neural network input using a core recurrent neural network to generate a neural network output; determining, from the neural network output, whether or not to end a currently invoked program and to return to a calling program from the set of programs; determining, from the neural network output, a next program to be called; determining, from the neural network output, contents of arguments to the next program to be called; receiving a representation of a current state of the environment; and generating a next neural network input from an embedding for the next program to be called and the representation of the current state of the environment.

Claims (58)

1. A neural network system for performing a machine learning task by invoking one or more programs selected from a set of programs that, when invoked, cause an environment to transition into a different state, the neural network system comprising:

a core neural network configured to receive a sequence of neural network inputs and to generate a sequence of neural network outputs;

a memory storing, for each program in the set of programs, a key identifying the program and an embedding for the program, wherein the embedding for the program is a collection of numeric values that represents the program, wherein the embeddings for the programs in the set of programs have been determined through training on a set of training data, and wherein one or more of the neural network inputs are generated based on an embedding for one of the programs in the set of programs; and

a subsystem configured to, for each neural network output:

determine, from the neural network output, whether or not to call a new program from the set of programs;

in response to determining to call a new program:

determine, from the neural network output, a next program to be called;

determine, from the neural network output, contents of arguments to the next program to be called;

invoke the next program with the contents of the arguments to the next program to cause the environment to transition into a current state;

obtain a representation of the current state of the environment; and

generate a next neural network input from the representation of the current state of the environment.

2. The neural network system of claim 1 , wherein generating the next neural network input comprises:

extracting a state encoding from the representation of the current state using a domain-specific encoder; and

generating the next neural network input using the state encoding.

3. The neural network system of claim 1 , wherein determining the next program to be called comprises:

determining, from the neural network output, a program key; and

selecting a program from the set of programs using the program key.

4. The neural network system of claim 3 , wherein selecting a program from the set of programs using the program key comprises:

selecting a program from the set of programs having a key that is most similar to the program key.

5. The neural network system of claim 1 , wherein the core neural network has been trained on training data that comprises execution traces.

6. The neural network system of claim 5 , wherein the embeddings for the programs in the set of programs have been determined during the training of the core neural network on the training data that comprises the execution traces.

7. The neural network system of claim 1 , wherein the core neural network comprises a recurrent neural network.

8. The neural network system of claim 1 , wherein the representation of the current state is an image.

9. The neural network system of claim 1 , wherein the subsystem is configured to continue generating next neural network inputs until an instance of a task has been performed.

10. The neural network system of claim 1 , wherein the machine learning task is a task performed for a user of a mobile device.

11. One or more non-transitory computer-readable media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for performing a machine learning task by invoking one or more programs selected from a set of programs that, when invoked, cause an environment to transition into a different state, the operations comprising:

storing, for each program in the set of programs, a key identifying the program and an embedding for the program, wherein the embedding for the program is a collection of numeric values that represents the program, wherein the embeddings for the programs in the set of programs have been determined through training on a set of training data, and wherein one or more of the neural network inputs are generated based on an embedding for one of the programs in the set of programs; and

processing a sequence of neural network inputs using a core neural network to generate a sequence of neural network outputs, wherein one or more of the neural network inputs are generated based on an embedding for one of the programs in the set of programs and wherein the processing comprises, for each neural network output:

determining, from the neural network output, whether or not to call a new program from the set of programs;

in response to determining to call a new program:

determining, from the neural network output, a next program to be called;

determining, from the neural network output, contents of arguments to the next program to be called;

invoking the next program with the contents of the arguments to the next program to cause the environment to transition into a current state;

obtaining a representation of the current state of the environment; and

generating a next neural network input from the representation of the current state of the environment.

12. A method for performing a machine learning task by invoking one or more programs selected from a set of programs that, when invoked, cause an environment to transition into a different state, the method comprising:

storing, for each program in the set of programs, a key identifying the program and an embedding for the program, wherein the embedding for the program is a collection of numeric values that represents the program, wherein the embeddings for the programs in the set of programs have been determined through training on a set of training data, and wherein one or more of the neural network inputs are generated based on an embedding for one of the programs in the set of programs; and

processing a sequence of neural network inputs using a core neural network to generate a sequence of neural network outputs, wherein one or more of the neural network inputs are generated based on an embedding for one of the programs in the set of programs and wherein the processing comprises, for each neural network output:

determining, from the neural network output, whether or not to call a new program from the set of programs;

in response to determining to call a new program:

determining, from the neural network output, a next program to be called;

determining, from the neural network output, contents of arguments to the next program to be called;

invoking the next program with the contents of the arguments to the next program to cause the environment to transition into a current state;

obtaining a representation of the current state of the environment; and

generating a next neural network input from the representation of the current state of the environment.

13. The method of claim 12 , wherein generating the next neural network input comprises:

extracting a state encoding from the representation of the current state using a domain-specific encoder; and

generating the next neural network input using the state encoding.

14. The method of claim 12 , wherein determining the next program to be called comprises:

determining, from the neural network output, a program key; and

selecting a program from the set of programs using the program key.

15. The method of claim 14 , wherein selecting a program from the set of programs using the program key comprises:

selecting a program from the set of programs having a key that is most similar to the program key.

16. The method of claim 12 , wherein the core neural network has been trained on training data that comprises execution traces.

17. The method of claim 16 , wherein the embeddings for the programs in the set of programs have been determined during the training of the core neural network on the training data that comprises the execution traces.

18. The method of claim 12 , wherein the core neural network comprises a recurrent neural network.

19. The method of claim 12 , wherein the representation of the current state is an image.

20. The method of claim 12 , wherein the machine learning task is a task performed for a user of a mobile device.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 6, 2025
From: DEEPMIND TECHNOLOGIES LIMITED
To: GDM HOLDING LLC
Reel/Frame 071498/0210 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 1, 2023
From: REED, SCOTT ELLISON; GOMES DE FREITAS, JOAO FERDINANDO
To: GOOGLE, INC.
Reel/Frame 065421/0583 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 1, 2023
From: GOOGLE, INC.
To: DEEPMIND TECHNOLOGIES LIMITED
Reel/Frame 065421/0654 →
Continuity (4)
Continuation 16859811 · Apr 27, 2020
Continuation 15349966 · Nov 11, 2016
Provisional Application 62285950 · Nov 12, 2015
Related Publication 20240177001A1 · May 30, 2024
References Cited (50)
US 9015093B1 · Commmons · 2015 [cited by applicant]
US 9037454B2 · Miolov et al. · 2015 [cited by applicant]
US 9037464B1 · Mikolov et al. · 2015 [cited by applicant]
US 20030200075A1 · Meng et al. · 2003 [cited by applicant]
US 20150066496A1 · Deoras et al. · 2015 [cited by applicant]
US 20170045894A1 · Canoy et al. · 2017 [cited by applicant]
CN 1682243A · 2005 [cited by applicant]
CN 101131695A · 2008 [cited by applicant]
CN 101625735A · 2010 [cited by applicant]
CN 104709288A · 2015 [cited by applicant]
Grefenstette, Edward, et al. “Learning to transduce with unbounded memory.” Advances in neural information processing systems 28 (2015). (Year: 2015). [cited by examiner]
Anderson. “Neural reuse: A fundamental organizational principle of the brain,” Behavioral and Brain Science, 33.04, Aug. 1, 2010, 69 pages. [cited by applicant]
Andre et al. “Programmable reinforcement learning agents,” Advances in Neural Information Processing Systems, Dec. 2001, 7 pages. [cited by applicant]
Aviles et al. “A recurrent neural network approach for 3d vision-based force estimation” IEEE, Oct. 2014, 6 pages. [cited by applicant]
Das et al. “Learning Context-free Grammars: Capabilities and Limitations of a Recurrent Neural Network with an External Stack Memory,” Advances in Neural Information Processing Systems 5, Jan. 1, 1993, 6 pages. [cited by applicant]
Dietterich. “Hierarchical reinforcement learning with the MAXQ value function decomposition,” Journal of Artificial Intelligence Research, 2000, 77 pages. [cited by applicant]
Donnarumma et al. “A programmer-interpreter neural network architecture for prefrontal cognitive control,” International Journal of Neural Systems, Sep. 2015, 25(6):1550017, 18 pages. [cited by applicant]
Donnarumma et al. “Programming in the brain: A neural network theoretical framework,” Connection Science, 24(2-3), Sep. 1, 2012, 22 pages. [cited by applicant]
Donnarumma, Francesco, et al. “A programmer-interpreter neural network architecture for prefrontal cognitive control.” International journal of neural systems 25.06 (2015): 1550017. [cited by applicant]
EP Summons to attend oral proceedings pursuant to Rule 115(1) EPC in European Appln. No. 16806344.4, dated Feb. 7, 2020, 7 pages. [cited by applicant]
Fidler et al. “3D object detection and viewpoint estimation with a deformable 3D cuboid model,” Advances in neural information processing systems, 2012, 9 pages. [cited by applicant]
Graves et al. “Neural Turing machines,” arXiv preprint arXiv:1410.5401, Oct. 20, 2014, 26 pages. [cited by applicant]
Greffenstette et al. “Learning to Transduce with Unbounded Memory,” Advances in Neural Information Processing Systems, Nov. 3, 2015, 14 pages. [cited by applicant]
Hochreiter et al. “Long short-term memory,” Neural computation, 9(8), Nov. 15, 1997, 46 pages. [cited by applicant]
Huang et al., “Research and Implementation of Intelligent Neural Network Programming Language,” Computer Engineering, 2007, 33(4):32-36 (with English abstract). [cited by applicant]
International Search Report and Written Opinion in International Application No. PCT/US2016/061682, mailed on Feb. 13, 2017, 13 pages. [cited by applicant]
Joulin et al. “Inferring algorithmic patterns with stack-augmented recurrent nets,” Advances in neural information processing systems, 2015, 9 pages. [cited by applicant]
Kaiser et al. “Neural gpus learn algorithms,” arXiv preprint arXiv:1511.08228, Nov. 25, 2015, 9 pages. [cited by applicant]
Kingma et al. “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, Dec. 22, 2014, 15 pages. [cited by applicant]
Kolter et al. “Hierarchical apprenticeship learning with application to quadruped locomotion,” Advances in Neural Information Processing Systems, 2007, 8 pages. [cited by applicant]
Kurach et al. “Neural random-access machines,” arXiv preprint arZiv: 1511.06392v.3, Feb. 9, 2016, 17 pages. [cited by applicant]
Mou et al. “Building program vector representations for deep learning,” arXiv preprint arXiv: 1409.3358, Sep. 11, 2014, 11 pages. [cited by applicant]
Neelakantan et al. “Neural programmer: Inducing latent programs with gradient descent,” arXiv preprint arXiv:1511.04834, Nov. 16, 2015, 18 pages. [cited by applicant]
Office Action in Chinese Appln. No. 201680066431.5, dated Aug. 2, 2021, 14 pages (with English translation). [cited by applicant]
OReilly et al. “Complementary learning systems,” Cognitive Science, 38(6), Aug. 1, 2014, 20 pages. [cited by applicant]
Reed et al. “Neural Programmer-Interpreters,” arXiv preprint arXiv 1511.06279v4, Feb. 29, 2016, 13 pages. [cited by applicant]
Rothkopf et al. “Modular inverse reinforcement learning for visuomotor behavior,” Biological Cybernetics, 107(4), Aug. 1, 2013, 24 pages. [cited by applicant]
Rumelhart et al. “Parallel distributed processing: Explorations in the microstructure of cognition”, MIT Press, 1986, 32 pages. [cited by applicant]
Schaul et al. “Universal value function approximators,” International Conference on Machines Learning, 2015, 9 pages. [cited by applicant]
Schmidhuber. “Learning to control fast-weight memories: An alternative to dynamic recurrent networks,” Neural Computation, 4(1), Jan. 1992, 9 pages. [cited by applicant]
Schneider et al. “Controlled and automatic processing: behavior, theory, and biological mechanism,” Cognitive Science, 27(3), Jan. 6, 2003, 35 pages. [cited by applicant]
Serban, et al. “Hierarchical neural network generative models for movie dialogues” arXiv, 2015, 11 pages. [cited by applicant]
Sutskever et al. “Sequence to sequence learning with neural networks,” In Advances in Neural Information Processing Systems, 2014, 9 pages. [cited by applicant]
Sutskever et al. “Using matrices to model symbolic relationship,” Advances in Neural Information Processing Systems, 2009, 8 pages. [cited by applicant]
Sutton et al. “Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning,” Artificial Intelligence, 112(1-2), Aug. 1999, 31 pages. [cited by applicant]
Van Rossum, et al “Python frequently asked questions” Institute for Theoretical Physics, 2014, 129 pages. [cited by applicant]
Vinyals et al. “Pointer networks,” Advances in Neural Information Processing Systems, 2015, 9 pages. [cited by applicant]
Zaremba et al. “Learning simple algorithms from examples,” arXiv preprint arXiv:1511.07275, 2015, 9 pages. [cited by applicant]
Zaremba et al. “Learning to execute,” arXiv preprint arXiv, Oct. 17, 2014, 25 pages. [cited by applicant]
Zaremba et al. “Reinforcement learning neural turing machines,” arXiv preprint arXiv: 1505.00521 362, May 2015, 13 pages. [cited by applicant]