IP Library › Granted Patent US 12,705,456
Granted Patent B2
US 12,705,456 · App. 17/914,066 · Granted Aug 11, 2026

Generating spatial embeddings by integrating agent motion and optimizing a predictive objective

Inventors: Benigno Uria-Martínez (London, GB); Andrea Banino (London, GB); Borja Ibarz Gabardos (London, GB); Vinicius Zambaldi (London, GB); Charles Blundell (London, GB)
Assignee: GDM Holding LLC
G06N3/044G06N3/045
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,705,456
App. No.
17/914,066
Filed
Sep 23, 2022
Granted
Aug 11, 2026
Kind
B2
Art Unit
2129
USPC
706/15
Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a spatial embedding neural network that is configured to process data characterizing motion of an agent that is interacting with an environment to generate spatial embeddings. In one aspect, a method comprises: processing data characterizing the motion of the agent in the environment at the current time step using a spatial embedding neural network to generate a current spatial embedding for the current time step; determining a predicted score and a target score for each of a plurality of slots in an external memory, wherein each slot stores: (i) a representation of an observation characterizing a state of the environment, and (ii) a spatial embedding; and determining an update to values of the set of spatial embedding neural network parameters based on an error between the predicted scores and the target scores.

Claims (65)

1 . A method performed by one or more computers for selecting actions to be performed by an agent to efficiently navigate an environment to perform a task, the method comprising:

controlling the agent to interact with the environment based on one or more spatial embeddings that encode spatial information for the environment, including repeatedly performing operations comprising:

processing current agent motion data using a spatial embedding neural network to generate a current spatial embedding; and

processing at least the current spatial embedding using an action selection neural network to generate a current action selection output; and

selecting a current action to be performed by the agent based on the current action selection output;

wherein the spatial embedding neural network has been trained using a training technique comprising, for each of a plurality of time steps:

processing agent motion data characterizing one or more of: a speed of the agent at the time step, an angular velocity of the agent at the time step, or a translational velocity of the agent at the time step, using the spatial embedding neural network to generate a current spatial embedding for the time step;

determining a predicted score and a target score for each of a plurality of slots in an external memory, wherein each slot corresponds to a respective previous time step and stores: (i) a representation of an observation comprising an image characterizing a state of the environment for the respective previous time step, and (ii) a spatial embedding for the respective previous time step generated based on agent motion data for the respective previous time step,

wherein the predicted score for each slot measures a similarity between: (i) the current spatial embedding, and (ii) the spatial embedding corresponding to the slot,

wherein the target score for each slot measures a similarity between: (i) a current observation comprising a current image characterizing the state of the environment at the time step, and (ii) the observation comprising the image corresponding to the slot; and

determining an update to values of a set of spatial embedding neural network parameters based on an error between the predicted scores and the target scores.

2 . The method of claim 1 , wherein the image is captured from a perspective of the agent at the time step.

3 . The method of claim 1 , wherein determining the target score for each slot in the external memory comprises:

obtaining respective embeddings of the current observation characterizing the current state of the environment and the observation corresponding to the slot; and

determining the target score based on a similarity measure between: (i) the embedding of the current observation characterizing the current state of the environment, and (ii) the embedding of the observation corresponding to the slot.

4 . The method of claim 3 , wherein obtaining the embedding of the current observation comprises processing the current observation using an embedding neural network.

5 . The method of claim 1 , wherein the error between the predicted scores and the target scores comprises a cross-entropy error between the predicted scores and the target scores.

6 . The method of claim 1 , further comprising determining an update to the spatial embeddings stored in the external memory based on the error between the predicted scores and the target scores.

7 . The method of claim 1 , wherein the spatial embedding neural network does not process the current observation to generate the current spatial embedding for the time step.

8 . The method of claim 1 , further comprising storing a representation of the current observation and the current spatial embedding in a slot in the external memory.

9 . The method of claim 1 , wherein the action selection output comprises a respective score for each action in a predetermined set of actions.

10 . The method of claim 9 , wherein selecting the current action to be performed by the agent based on the current action selection output comprises selecting an action having a highest score.

11 . The method of claim 1 , wherein the action selection neural network is configured to additionally processes a goal spatial embedding that was generated by the spatial embedding neural network at a time step when the agent was located in a goal location in the environment.

12 . The method of claim 1 , wherein the spatial embedding neural network comprises a recurrent neural network, and wherein generating the current spatial embedding for the time step comprises:

processing: (i) the agent motion data characterizing one or more of: a speed of the agent at the time step, an angular velocity of the agent at the time step, or a translational velocity of the agent at the time step, and (ii) an output of the spatial embedding neural network at a preceding time step, using the spatial embedding neural network to update a hidden state of the spatial embedding neural network, wherein the updated hidden state defines the current spatial embedding.

13 . The method of claim 12 , further comprising determining an output of the spatial embedding neural network for the time step, comprising:

identifying the updated hidden state of the spatial embedding neural network as the output of the spatial embedding neural network for the time step.

14 . The method of claim 12 , further comprising determining an output of the spatial embedding neural network for the time step, comprising:

determining a respective weight value for each slot in the external memory that characterizes a similarity between: (i) the current observation characterizing the state of the environment at the time step, and (ii) the observation corresponding to the slot;

determining a correction embedding as a linear combination of the spatial embeddings corresponding to the slots in the external memory, wherein each spatial embedding is weighted by the respective weight value; and

determining the output of the spatial embedding neural network based on: (i) the updated hidden state of the spatial embedding neural network, and (ii) the correction embedding.

15 . The method of claim 1 , further comprising:

processing agent motion data characterizing one or more of: a speed of the agent at the time step, an angular velocity of the agent at the time step, or a translational velocity of the agent at the time step using an integrated embedding neural network having a set of integrated embedding neural network parameters to generate a current integrated embedding for the time step;

determining a predicted score and a target score for each of a plurality of slots in an additional external memory, wherein each slot stores: (i) a spatial embedding, and (ii) an integrated embedding,

wherein the predicted score for each slot measures a similarity between: (i) the current integrated embedding, and (ii) the integrated embedding corresponding to the slot,

wherein the target score for each slot measures a similarity between: (i) the current spatial embedding, and (ii) the spatial embedding corresponding to the slot; and

determining an update to values of the set of integrated embedding neural network parameters based on an error between the predicted scores and the target scores for the slots in the additional external memory.

16 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for selecting actions to be performed by an agent to efficiently navigate an environment to perform a task, the operations comprising:

controlling the agent to interact with the environment based on one or more spatial embeddings that encode spatial information for the environment, including repeatedly performing operations comprising:

processing current agent motion data using a spatial embedding neural network to generate a current spatial embedding; and

processing at least the current spatial embedding using an action selection neural network to generate a current action selection output; and

selecting a current action to be performed by the agent based on the current action selection output;

wherein the spatial embedding neural network has been trained using a training technique comprising, for each of a plurality of time steps:

processing agent motion data characterizing one or more of: a speed of the agent at the time step, an angular velocity of the agent at the time step, or a translational velocity of the agent at the time step, using the spatial embedding neural network to generate a current spatial embedding for the time step;

determining a predicted score and a target score for each of a plurality of slots in an external memory, wherein each slot corresponds to a respective previous time step and stores: (i) a representation of an observation comprising an image characterizing a state of the environment for the respective previous time step, and (ii) a spatial embedding for the respective previous time step generated based on agent motion data for the respective previous time step,

wherein the predicted score for each slot measures a similarity between: (i) the current spatial embedding, and (ii) the spatial embedding corresponding to the slot,

wherein the target score for each slot measures a similarity between: (i) a current observation comprising a current image characterizing the state of the environment at the time step, and (ii) the observation comprising the image corresponding to the slot; and

determining an update to values of a set of spatial embedding neural network parameters based on an error between the predicted scores and the target scores.

17 . A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for selecting actions to be performed by an agent to efficiently navigate an environment to perform a task, the operations comprising:

controlling the agent to interact with the environment based on one or more spatial embeddings that encode spatial information for the environment, including repeatedly performing operations comprising:

processing current agent motion data using a spatial embedding neural network to generate a current spatial embedding; and

processing at least the current spatial embedding using an action selection neural network to generate a current action selection output; and

selecting a current action to be performed by the agent based on the current action selection output;

wherein the spatial embedding neural network has been trained using a training technique comprising, for each of a plurality of time steps:

processing agent motion data characterizing one or more of: a speed of the agent at the time step, an angular velocity of the agent at the time step, or a translational velocity of the agent at the time step, using the spatial embedding neural network to generate a current spatial embedding for the time step;

determining a predicted score and a target score for each of a plurality of slots in an external memory, wherein each slot corresponds to a respective previous time step and stores: (i) a representation of an observation comprising an image characterizing a state of the environment for the respective previous time step, and (ii) a spatial embedding for the respective previous time step generated based on agent motion data for the respective previous time step,

wherein the predicted score for each slot measures a similarity between: (i) the current spatial embedding, and (ii) the spatial embedding corresponding to the slot,

wherein the target score for each slot measures a similarity between: (i) a current observation comprising a current image characterizing the state of the environment at the time step, and (ii) the observation comprising the image corresponding to the slot; and

determining an update to values of a set of spatial embedding neural network parameters based on an error between the predicted scores and the target scores.

18 . The system of claim 17 , wherein the image is captured from a perspective of the agent at the time step.

19 . The system of claim 17 , wherein determining the target score for each slot in the external memory comprises:

obtaining respective embeddings of the current observation characterizing the current state of the environment and the observation corresponding to the slot; and

determining the target score based on a similarity measure between: (i) the embedding of the current observation characterizing the current state of the environment, and (ii) the embedding of the observation corresponding to the slot.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 6, 2025
From: DEEPMIND TECHNOLOGIES LIMITED
To: GDM HOLDING LLC
Reel/Frame 071498/0210 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 23, 2023
From: URIA-MARTÍNEZ, BENIGNO; BANINO, ANDREA; IBARZ GABARDOS, BORJA; ZAMBALDI, VINICIUS; BLUNDELL, CHARLES
To: DEEPMIND TECHNOLOGIES LIMITED
Reel/Frame 062787/0802 →
Continuity (2)
Provisional Application 63025477 · May 15, 2020
Related Publication 20230124261A1 · Apr 20, 2023
References Cited (71)
US 20200051252A1 · Brown et al. · 2020 [cited by applicant]
US 20200134425A1 · Chen et al. · 2020 [cited by applicant]
CN 110088774A · 2019 [cited by applicant]
WO WO2019170905A1 · 2019 [cited by examiner]
Beeching et al., Ego Map: Projective mapping and structured egocentric memory for Deep RL. arXiv preprint arXiv:2002.02286. Jan. 24, 2020. [cited by examiner]
Zhang, Jingwei, et al. “Neural slam: Learning to explore with external memory.” arXiv preprint arXiv:1706.09520 (Year: 2017). [cited by examiner]
Oh, Junhyuk et al., “Control of memory, active perception, and action in minecraft.” International conference on machine learning. PMLR (Year: 2016). [cited by examiner]
Office Action in European Appln. No. 21726089.2, dated Mar. 18, 2024, 11 pages. [cited by applicant]
Alexander et al., “Egocentric boundary vector tuning of the retrosplenial cortex,” Science Advances, Feb. 2020, 6(8):eaaz2322. [cited by applicant]
Banino et al., “Vector-based navigation using grid-like representations in artificial agents,” Nature, May 2018, available online May 9, 2018, 557(7705):429-433 (21 pages including the supplemental material). [cited by applicant]
Barry et al., “Experience-dependent rescaling of entorhinal grids,” Nature Neuroscience, Jun. 2007, available online May 7, 2007, 10(6):682-684, 3 pages. [cited by applicant]
Barry et al., “The boundary vector cell model of place cell firing and spatial memory,” Reviews in the Neurosciences, 2006, 17(1-2):71-97. [cited by applicant]
Beattie et al., “Deepmind lab,” CoRR, Submitted on Dec. 12, 2016, arXiv:1612.03801v1, 11 pages. [cited by applicant]
Bicanski et al., “A neural-level model of spatial memory and imagery,” Sep. 2018, eLife, 7:e33752, 45 pages. [cited by applicant]
Blum et al., “A model of spatial map formation in the hippocampus of the rat ,” Neural Computation, Jan. 1996, 8(1):85-93. [cited by applicant]
Byrne et al., “Remembering the past and imagining the future: a neural model of spatial memory and imagery,” Psychological Review, Apr. 2007, 114(2):340-375. [cited by applicant]
Cueva et al. “Emergence of grid-like representations by training recurrent neural networks to perform spatial localization,” CoRR, Submitted on May 21, 2018, arXiv:1803.07770v1, 19 pages. [cited by applicant]
Dordek et al., “Extracting grid cell characteristics from place cell inputs using non- negative principal component analysis,” eFlie, May 2016, 5:e10094, 36 pages. [cited by applicant]
Eichenbaum, “Hippocampus: cognitive processes and neural representations that underlie declarative memory,” Neuron, Sep. 2004, 44(1):109-120. [cited by applicant]
Engel et al., “LSD-SLAM; large-scale direct monocular SLAM,” European Conference on Computer Vision, 2014, pp. 834-849. [cited by applicant]
Espeholt, et al., “Impala: scalable distributed deep-rl with importance weighted actorlearner architectures,” CoRR, Submitted on Feb. 5, 2018, arXiv:1802.01561v1, 21 pages. [cited by applicant]
Foster et al., “Reverse replay of behavioural sequences in hippocampal place cells during the awake state,” Nature, Mar. 2006, available online Feb. 12, 2006, 440(7084):680-683. [cited by applicant]
Franzius et al., “Slowness and sparseness lead to place, head-direction, and spatial-view cells,”, PLoS Computational Biology, 2007, 3(8):e166, 18 pages. [cited by applicant]
Fríston et al., “Predictive coding under the free-energy principle,” Philosophical Transactions of the Royal Society B: Biological Sciences, May 2009, 364(1521):1211-1221. [cited by applicant]
Fuhs et al., “Synaptic learning models of map separation in the hippocampus,” Neurocomputing, Jun. 2000, 32-33:379-384. [cited by applicant]
Gothard et al., “Binding of hippocampal CAI neural activity to multiple reference frames in a landmark-based navigation task,” Journal of Neuroscience, Jan. 1996, 16(2):823-835. [cited by applicant]
Hafting et al., “Microstructure of a spatial map in the entorhinal cortex,” Nature, Aug. 2005, available online Jun. 19, 2005, 436(7052):801-806. [cited by applicant]
Hartley et al., “Modeling place fields in terms of the cortical inputs to the hippocampus,” Hippocampus, 2000, 10(4):369-379. [cited by applicant]
Hassabis et al., “The construction system of the brain,” Philosophical Transactions of the Royal Society B: Biological Sciences, May 2009, 364(1521):1263-1271. [cited by applicant]
Hayman et al., “Context-specific acquisition of location discrimination by hippocampal place cells,” European Journal of Neuroscience, Nov. 2003, 18(10):2825-2834. [cited by applicant]
Hinman et al., “Neuronal representation of environmental boundaries in egocentric coordinates,” Nature Communications, Jun. 2019, 10:2772, 8 pages. [cited by applicant]
Hinton et al., “Distributed representations,” Carnegie-Mellon University, Oct. 1984, 34 pages. [cited by applicant]
International Preliminary Report on Patentability in International Appln. No. PCT/EP2021/062704, dated Nov. 24, 2022, 11 pages. [cited by applicant]
International Search Report and Written Opinion in International Appln. No. PCT/EP2021/062704, dated Aug. 12, 2021, 17 pages. [cited by applicant]
Jonschowski et al., “PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations,” CoRR, arXiv:1705.09805v3, Jul. 24, 2017, 7 pages. [cited by applicant]
Kim et al., “Ring attractor dynamics in the [cited by applicant]
Klein et al., “Parallel tracking and mapping for small AR workspaces,” Proceedings of the 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality (IEEE Computer Society, 2007), pp. 1-10. [cited by applicant]
Kumaran et al., “An unexpected sequence of events: mismatch detection in the human hippocampus,” PLoS Biology, Dec. 2006, 4(12):e424, 11 pages. [cited by applicant]
Lee et al., “Memory of sequential experience in the hippocampus during slow wave sleep,” Neuron, Dec. 2002, 36(6):1183-1194. [cited by applicant]
Leutgeb et al., “Independent codes for spatial and episodic memory in hippocampal neuronal ensembles,” Science, Jul. 2005, 309(5734):619-623. [cited by applicant]
Lever et al., “Boundary vector cells in the subiculum of the hippocampal formation,” Journal of Neuroscience, Aug. 2009, 29(31):9771-9777. [cited by applicant]
Lisman et al., “Prediction, sequences and the hippocampus,” Philosophical Transactions of the Royal Society B: Biological Sciences, May 2009, 364(1521):1193-1201. [cited by applicant]
Morris et al., “Spatial localization does not require the presence of local cues,” Learning and Motivation, May 1981, 12(2):239-260. [cited by applicant]
Moser et al., “Place cells, grid cells, and memory,” Cold Spring Harbor Perspectives in Biology, 2015, 7:a021808, 16 pages. [cited by applicant]
Muller et al., “The effects of changes in the environment on the spatial firing of hippocampal complex-spike cells,” Journal of Neuroscience, Jul. 1987, 7(7):1951-1968. [cited by applicant]
O'Craven et al., “Mental imagery of faces and places activates corresponding stiimulus-specific brain regions,” Journal of Cognitive Neuroscience, Nov. 2000, 12(6):1013-1023. [cited by applicant]
O'Keefe et al., “Geometric determinants of the place fields of hippocampal neurons,” Nature, May 1996, 381(6581):425-428. [cited by applicant]
O'Keefe et al., “The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat,” Brain Research 34(1):171-175. [cited by applicant]
Pfeiffer et al., “Hippocampal place-cell sequences depict future paths to remembered goals,” Nature, May 2013, available online Apr. 17, 2013, 497(7447):74-79, (8 pages including the supplemental material). [cited by applicant]
Poulter et al., “Vector trace cells in the subiculum of the hippocampal formation,” bioRxiv, Oct. 2019, 43 pages. [cited by applicant]
Rao et al., “Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects,” Nature Neuroscience, Jan. 1999, 2(1):79- 87. [cited by applicant]
Raudies et al., “Modeling boundary vector cell firing given optic flow as a cue,” PLoS Computational Biology, Jun. 2012, 9(6) e1002553, 17 pages. [cited by applicant]
Redish et al., “A coupled attractor model of the rodent head direction system,” Network: Computation in Neural Systems, 1996, 7(4):671-685. [cited by applicant]
Rivard et al., “Representation of objects in space by two classes of hippocampal pyramidal cells,” The Journal of General Physiology, Jul. 2004, 124(1):9-25. [cited by applicant]
Savinov et al., “Episodic curiosity through reachability,” CoRR, Aug. 6, 2019, arXiv:1810.02274v5, 27 pages. [cited by applicant]
Schiller et al., “Memory and space: towards an understanding of the cognitive map,” Journal of Neuroscience, Oct. 2015, 35(41):13904-13911. [cited by applicant]
Seelig et al., “Neural dynamics for landmark orientation and angular path integration,” Nature, May 2015, 521(7551):186-191. [cited by applicant]
Sharp et al., “The anatomical and computational basis of the rat head-direction cell signal,” Trends in Neurosciences, May 2001, 24(5):289-294. [cited by applicant]
Sharp, “Computer simulation of hippocampal place cells,” Psychobiology, 1991, 19(2):103-115. [cited by applicant]
Skaggs et al., “A model of the neural basis of the rat's sense of direction,” Advances in Neural Information Processing Systems, 1995, pp. 173-180. [cited by applicant]
Solstad et al., “From grid cells to place cells: a mathematical model,” Hippocampus, 2006, 16(12):1026-1031. [cited by applicant]
Solstad et al., “Representation of geometric borders in the entorhinal cortex,” Science, Dec. 2008, 322(5909):1865-1868. [cited by applicant]
Stachenfeld et al., “The hippocampus as a predictive map,” Nature Neuroscience, Nov. 2017, available online Oct. 2017, 20(11):1643-1653 (13 pages including the supplemental material). [cited by applicant]
Sussillo et al., “Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks,” Neural Computation, Mar. 2013, 25(3):626-649. [cited by applicant]
Taube et al., “Head-direction cells recorded from the postsubiculum in freely moving rats. II. Effects of environmental manipulations,” Journal of Neuroscience, Feb. 1990, 10(2):436-447. [cited by applicant]
Wang et al., “Egocentric coding of external items in the lateral entorhinal cortex,” Science, Nov. 2018, 362(6417):945-949. [cited by applicant]
Whittington et al., “Generalisation of structural knowledge in the hippocampal-entorhinal system,” Advances in Neural Information Processing Systems, 2018, 12 pages. [cited by applicant]
Zhang et al., “Representation of spatial orientation by the intrinsic dynamics of the head-direction cell ensemble: a theory,” Journal of Neuroscience, Mar. 1996, 16(6):2112-2126. [cited by applicant]
Zipser, “A computational model of hippocampal place fields,” Behavioral Neuroscience, 1985, 99(5):1006-1018. [cited by applicant]
Office Action in Chinese Appln. No. 202180023280.6, mailed on May 15, 2025, 10 pages (with English translation). [cited by applicant]
Ying et al., “Survey on Image Classification Technology Based on Small Sample Learning,” Acta Automatica Sinca, Feb. 2021, 47(2):297-315. [cited by applicant]