IP Library › Granted Patent US 12,299,575

Granted Patent B2

US 12,299,575 · App. 17/093,373 · Granted May 13, 2025

Augmenting neural networks with external memory

Inventors: Alexander Benjamin Graves (London, GB); Ivo Danihelka (London, GB); Timothy James Alexander Harley (London, GB); Malcolm Kevin Campbell Reynolds (London, GB); Gregory Duncan Wayne (London, GB)

Assignee: DeepMind Technologies Limited

G06N3/082 G06N3/04 G06N3/044 G06N3/063

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12,299,575
App. No.: 17/093,373
Granted: May 13, 2025
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for augmenting neural networks with an external memory. One of the systems includes a memory interface subsystem that is configured to perform operations comprising determining a respective content-based weight for each of a plurality of locations in an external memory; determining a respective allocation weight for each of the plurality of locations in the external memory; determining a respective final writing weight for each of the plurality of locations in the external memory from the respective content-based weight for the location and the respective allocation weight for the location; and writing data defined by the write vector to the external memory in accordance with the final writing weights.

Claims (93)

1. An augmented neural network system for processing a sequence of system inputs to generate a sequence of system outputs, the augmented neural network system comprising:

a neural network, wherein the neural network is configured to receive a sequence of neural network inputs and to process each neural network input to generate a neural network output from the neural network input, wherein the neural network output comprises:

a system output portion,

a read portion,

a write key, and

a write vector;

an external memory; and

a memory interface subsystem, wherein the memory interface subsystem is configured to perform operations comprising, for each of the neural network outputs:

providing an output derived from the system output portion as a system output in the sequence of system outputs;

determining a respective allocation weight for each of the plurality of locations in the external memory based on how available each of the plurality of locations in the external memory is for a modification of the data stored in the location, comprising:

determining a respective usage value for each of the locations that represents a degree to which the availability of the location is being used, comprising, for each of the locations, increasing a previous usage value for the location based on a previous final writing weight used to write to the location for a previous neural network output, and

determining the respective allocation weights for the locations based at least in part on the respective usage values for the locations;

determining a respective final writing weight for each of the plurality of locations in the external memory from the respective allocation weight for the location;

writing data defined by the write vector to the external memory in accordance with the final writing weights;

determining one or more sets of reading weights for each of the plurality of locations in the external memory from the read portion;

reading data from the external memory in accordance with the sets of reading weights; and

combining the data read from the external memory with a next system input in the sequence of system inputs to generate a next neural network input in the sequence of neural network inputs.

2. The augmented neural network system of claim 1 , wherein determining the respective allocation weights for the locations based at least in part on the respective usage values for the locations comprises:

ordering the locations according to the usage values; and

determining the respective allocation weights for the locations based on the ordering and on the usage values.

3. The augmented neural network system of claim 1 , wherein the neural network output comprises a set of free gates that define an extent to which each of the plurality of locations can be freed when writing to the external memory, and wherein determining the respective usage value for each of the locations comprises:

modifying the increased previous usage value for the location based on the extent to which the location can be freed to generate the respective usage value for the location.

4. The augmented neural network system of claim 2 , wherein ordering the locations according to the usage values comprises ordering the locations in ascending order of usage.

5. The augmented neural network system of claim 4 , wherein determining the respective allocation weight for each of the locations comprises:

determining an initial allocation weight for the location by subtracting the usage value for the location from one; and

modifying the initial allocation weight based on the usage values for other locations in other positions in the ordering.

6. The augmented neural network system of claim 1 , wherein the neural network output comprises:

an allocation gate that governs interpolation between (i) content-based weights for the plurality of locations and (ii) the allocation weights, and

a write gate that governs a write strength of the write to the external memory, and wherein determining a respective final writing weight for each of the plurality of locations comprises:

interpolating between the content-based weight for the location and the allocation weight for the location in accordance with the allocation gate, and

gating a result of the interpolation in accordance with the write gate to determine the final writing weight for the location.

7. The augmented neural network system of claim 1 , wherein the neural network output further comprises an erase vector, and wherein the operations further comprise, for each of the neural network outputs:

erasing from the external memory in accordance with the final writing weights and the erase vector.

8. An augmented neural network system for processing a sequence of system inputs to generate a sequence of system outputs, the augmented neural network system comprising:

a system output portion,

a write portion,

a write vector, and

a respective read key for each of one or more read heads;

an external memory; and

a memory interface subsystem, wherein the memory interface subsystem is configured to perform operations comprising, for each of the neural network outputs:

providing an output derived from the system output portion as a system output in the sequence of system outputs;

determining a respective writing weight for each of a plurality of locations in the external memory from the write portion of the neural network output;

writing data defined by the write vector to the external memory in accordance with the respective writing weights;

maintaining a temporal link matrix that tracks a history of writing weights for previous writing operations performed on the external memory by the memory interface subsystem, wherein each element in the temporal link matrix represents a degree to which a corresponding first location was written to after a corresponding second location by a previous writing operation for a previous neural network output;

updating the temporal link matrix based on the writing weights for the neural network output;

for each of one or more read heads:

determining one or more respective history weights for each of the plurality of locations in the external memory from the updated temporal link matrix,

determining a respective final reading weight for each of the plurality of locations from the one or more history weights, and

reading data from the external memory in accordance with the final reading weights; and

combining the data read from the external memory with a next system input in the sequence of system inputs to generate a next neural network input in the sequence of neural network inputs.

9. The augmented neural network system of claim 8 , the operations further comprising, for each of the neural network outputs:

determining one or more sets of erasing weights for each of the plurality of locations in the external memory; and

erasing data from the external memory in accordance with the sets of erasing weights.

10. The augmented neural network system of claim 8 , wherein the neural network output comprises a respective read mode vector for each read head that governs an interpolation between (i) the one or more history weights and (ii) content-based reading weights for the plurality of locations, and wherein determining a respective final reading weight for each of the plurality of locations from the one or more history weights comprises:

interpolating between the one or more history weights for the location and the content-based reading weight for the location in accordance with the read mode vector.

11. The augmented neural network system of claim 8 , wherein determining one or more respective history weights for each of the plurality of locations in the external memory comprises:

determining a respective backward history weight for each of the plurality of locations from the updated temporal link matrix.

12. The augmented neural network system of claim 11 , wherein determining a respective backward history weight for each of the plurality of locations from the updated temporal link matrix comprises:

performing a matrix multiplication between the updated temporal link matrix and a vector derived from the final reading weights for the read head for a previous neural network output.

13. The augmented neural network system of claim 8 , wherein determining one or more respective history weights for each of the plurality of locations in the external memory comprises:

determining a respective forward history weight for each of the plurality of locations from a transpose of the updated temporal link matrix.

14. The augmented neural network system of claim 13 , wherein determining a respective forward history weight for each of the plurality of locations from the updated temporal link matrix comprises:

performing a matrix multiplication between the transpose of the updated temporal link matrix and a vector derived from the final reading weights for the read head for a previous neural network output.

15. The augmented neural network system of claim 8 , wherein updating the temporal link matrix comprises:

determining a respective precedence weight for each of the plurality of locations that represents a degree to which the location was the last location written to; and

updating the temporal link matrix using the precedence weighting for each of the plurality of locations and the writing weights for the neural network output.

16. A method of processing a sequence comprising a plurality of system inputs to generate a sequence comprising a plurality of system outputs, the method comprising, for each of the system inputs:

obtaining data previously read from an external memory during processing of a preceding system input;

combining the data previously read from the external memory with the system input to generate a neural network input;

providing the neural network input as input to a neural network, wherein the neural network is configured to process the neural network input to generate a neural network output from the neural network input, wherein the neural network output comprises:

a system output portion,

a read portion,

a write key, and

a write vector;

determining the respective allocation weights for the locations based at least in part on the respective usage values for the locations;

determining a respective final writing weight for each of the plurality of locations in the external memory from the respective allocation weight for the location;

writing data defined by the write vector to the external memory in accordance with the final writing weights;

determining one or more sets of reading weights for each of the plurality of locations in the external memory from the read portion;

reading data from the external memory in accordance with the sets of reading weights; and

providing an output derived from the system output portion as a system output in the sequence of system outputs.

17. The method of claim 16 , wherein determining the respective allocation weights for the locations based at least in part on the respective usage values for the locations comprises:

ordering the locations according to the usage values; and

determining the respective allocation weights for the locations based on the ordering and on the usage values.

18. The method of claim 17 , wherein ordering the locations according to the usage values comprises ordering the locations in ascending order of usage.

19. The method of claim 18 , wherein determining the respective allocation weight for each of the locations comprises:

determining an initial allocation weight for the location by subtracting the usage value for the location from one; and

modifying the initial allocation weight based on the usage values for other locations in other positions in the ordering.

20. The method of claim 16 , wherein the neural network output further comprises an erase vector, and wherein the method further comprises, for each of the neural network outputs:

erasing from the external memory in accordance with the final writing weights and the erase vector.

Assignments (3)

ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 6, 2025

From: DEEPMIND TECHNOLOGIES LIMITED

To: GDM HOLDING LLC

Reel/Frame 071498/0210 →

ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 10, 2020

From: GRAVES, ALEXANDER BENJAMIN; DANIHELKA, IVO; HARLEY, TIMOTHY JAMES ALEXANDER; REYNOLDS, MALCOLM KEVIN CAMPBELL; WAYNE, GREGORY DUNCAN

To: GOOGLE INC.

Reel/Frame 054327/0234 →

ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 10, 2020

From: GOOGLE INC.

To: DEEPMIND TECHNOLOGIES LIMITED

Reel/Frame 054327/0677 →

Continuity (3)

Continuation 15374974 · Dec 9, 2016

Provisional Application 62265912 · Dec 10, 2015

Related Publication 20210117801A1 · Apr 22, 2021

References Cited (62)

US 5167006A · Furuta · 1992 [cited by examiner]

US 6092018A · Puskorius et al. · 2000 [cited by applicant]

US 20050041453A1 · Brazis · 2005 [cited by examiner]

US 20090172250A1 · Allen · 2009 [cited by examiner]

US 20140180694A1 · Fastow et al. · 2014 [cited by applicant]

US 20150220833A1 · Le · 2015 [cited by applicant]

CN 101076806 · 2007 [cited by applicant]

CN 101253493 · 2008 [cited by applicant]

CN 103617235 · 2014 [cited by applicant]

CN 103824291 · 2014 [cited by applicant]

CN 104657776 · 2015 [cited by applicant]

CN 105144164 · 2015 [cited by applicant]

WO WO0072256 · 2000 [cited by applicant]

Office Action in European Appln. No. 16820464.2, dated May 27, 2021, 10 pages. [cited by applicant]

Amari, “Characteristics of sparsely encoded associative memory,” Neural Networks, 1989, 2.6:451-457. [cited by applicant]

Bahdanau et al. “Neural machine translation by jointly learning to align and translate,” CoRR, 2014, arXiv preprint arXiv, 1409.0473. [cited by applicant]

Bengio et al. “Curriculum learning,” In Proceedings of the 26 [cited by applicant]

Bollacker et al., “Freebase: a collaboratively created graph database for structuring human knowledge,” Proceedings of the structure 2008 ACM SIGMOD international conference on Management of DATA. ACM, 2008, pp. 1247-12… [cited by applicant]

Bottou., “From machine learning to machine reasoning,” Machine learning, 2011, 94.2:133-149. [cited by applicant]

Chan et al., “Listen, attend and spell,” CoRR, 2015, arXiv preprint arXiv, 1508.01211. [cited by applicant]

Das et al., “Learning Context-free Grammars: Capabilities and Limitations of a Recurrent Neural Network with an External Stack Memory,” Advances in Neural Information Processing Systems 5., Jan. 1, 1993, 6 pages. [cited by applicant]

Daume et al., “Search-based structured prediction,” Machine Learning, Jun. 1, 2009, 75(3):297-325. [cited by applicant]

Dean et al., “Large scale distributed deep networks,” Advances in neural information processing systems, 2012, pp. 1223-1231. [cited by applicant]

Fusi et al., “Cascade models of synaptically stored memories,” Neuron 45.4 2005, pp. 599-611. [cited by applicant]

Ganguli et al., “Memory traces in dynamical systems,” Proceedings of the National Academy of Sciences, 2008, 105.48:18970-18975. [cited by applicant]

Graves et al., “Neural turing machines” CoRR, 2014, arXiv preprint arXiv, 1410.5401, 26 pages. [cited by applicant]

Graves et al., “Generating sequences with recurrent neural networks,” CoRR, Aug. 2014, arXiv preprint arXiv, 1308.0850. [cited by applicant]

Graves et al., “Speech recognition with deep recurrent neural networks,” IEEE international conference on acoustics, speech and signal processing, 2013, pp. 6645-6649. [cited by applicant]

Graves., “Generating sequences with recurrent neural networks,” CoRR, Aug. 2014, arXiv preprint arXiv, 1308.0850. [cited by applicant]

Grefenstette et al., “Learning to Transduce with Unbounded memory,” CoRR, Nov. 2015, arXiv preprint arXiv 1506.02516, 14 pages. [cited by applicant]

Gregor et al., “DRAW: A recurrent neural network for image generation,” CoRR, 2015, arXiv prepring arXiv, 1502.04623. [cited by applicant]

Hinton., “Learning distributed representations of concepts,” Proceedings of the eighth annual conference of the cognitive science society, 1986, 1:12. [cited by applicant]

Hochreiter et al., “Long short-term memory,” Neural computation, 1997, 9.8:1735-1780. [cited by applicant]

Hopfield., “Neural networks and physical systems with emergent collective computational abilities,” Proceeding of the national academy of sciences, 1982, 79.8:2554-2558. [cited by applicant]

International Search Report and Written Opinion in International Application No. PCT/US2016/066020, dated Mar. 15, 2017, 19 pages. [cited by applicant]

Joulin et al., “Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets,” CoRR, Jun. 2015, arXiv preprint arXiv1503.01007, 10 pages. [cited by applicant]

JP Office Action in Japanese Appln. No. 2018-530744, dated Sep. 9, 2019, 6 pages (with English translation). [cited by applicant]

Kakade et al., “Approximately optimal approximate reinforcement learning,” ICML, Jul. 8, 2002, 2:267-274. [cited by applicant]

KR Office Action in Korean Appln. No. 10-2018-7017730, dated Dec. 12, 2019, 23 pages (with English Translation). [cited by applicant]

KR Office Action in Korean Appln. No. 10-2018-7017730, dated May 7, 2020, 10 pages (with English translation). [cited by applicant]

Kriete et al., “Indirection and symbol-like processing in the prefrontal cortex and basal ganglia,” Proceedings of the National Academy of Sciences, 2013, 110.41:16390-16395. [cited by applicant]

Krizhevsky et al., “Imagenet classification with deep convolutional neural networks,” In Advances in neural information processing systems, 2012, pp. 1097-1105. [cited by applicant]

Kurach et al., “Neural Random-Access Machines,” CoRR, Nov. 2015, arXiv preprint arXiv1511.06392v1, 13 pages. [cited by applicant]

Miller, “Wordnet: a lexical database for English,” Communications of the ACM, 1995, 38(11):39-41. [cited by applicant]

Minsky, “Steps toward artificial intelligence,” Proceedings of the IRE, 1961, 49.1:8-30. [cited by applicant]

Mnih et al., “Human-level control through deep reinforcement learning,” Nature, Feb. 26, 2015, 518:529-533. [cited by applicant]

Rae et al., “Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes,” CoRR, Oct. 2016, arXiv preprint arXiv1610.09027, 17 pages. [cited by applicant]

Ross et al., “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning,” AISTATS, Apr. 11, 2011, 1(2):6. [cited by applicant]

Schulman et al., “High-dimensional continuous control using generalized advantage estimation,” CoRR, 2015, arXiv preprint arXIV, 1506.02438. [cited by applicant]

Sutskever et al., “Sequence to sequence learning with neural networks,” In Advances in neural information processing systems, 2014, pp. 3104-3112. [cited by applicant]

Van der Maaten et al., “Visualizing data using t-SNE,” Journal of Machine Learning Research, Nov. 9, 2008, pp. 2579-2605. [cited by applicant]

Vinyals et al., “Pointer networks,” Advances in Neural Information Processing Systems, 2015, pp. 2692-2700. [cited by applicant]

Wawrzyński, “Real-time reinforcement learning by sequential actor-critics and experience replay,” Neural Networks, Dec. 31, 2009 22(10):1484-1497. [cited by applicant]

Werbos, “Backpropagation through time: what it does and how to do it,” Proceedings of the IEEE, 1990, 78(10):1550-1560. [cited by applicant]

Weston et al., “Memory Networks,” CoRR, 2014, arXiv preprint arXiv, 1410.3916. [cited by applicant]

Wilson et al., “Dynamic storage allocation: A survey and critical review,” Memory Management. Springer Berlin Heidelberg, 1995, 78 pages. [cited by applicant]

Winograd, “Procedures as a representation for data in a computer program for understanding natural language,” Mass. Inst. of Tech. Cambridge Project Mac., 1971, MAC-TR-84. [cited by applicant]

Zaremba et al., “Learning to execute,” CoRR, 2014, arXiv preprint arXiv, 1410.4615. [cited by applicant]

Zhang et al., “Structured Memory for Neural Turing Machines,” CoRR, Oct. 2015, arXiv:1510.03931v3, 4 pages. [cited by applicant]

Office Action in Chinese Appln. No. 201680072537.6, dated Sep. 29, 2021, 31 pages (with English translation). [cited by applicant]

Liu et al., “Sparse convolutional neural networks,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 806-814. [cited by applicant]

Office Action in Chinese Appln. No. 201680072537.6, dated Jun. 21, 2022, 8 pages (with English translation). [cited by applicant]