IP Library › Granted Patent US 12,499,359
Granted Patent B2
US 12,499,359 · App. 17/046,963 · Granted Dec 16, 2025

Graph neural networks representing physical systems

Inventors: Martin Riedmiller (Balgheim, DE); Raia Thais Hadsell (London, GB); Peter William Battaglia (London, GB); Joshua Merel (London, GB); Jost Tobias Springenberg (London, GB); Alvaro Sanchez (London, GB); Nicolas Manfred Otto Heess (London, GB)
Assignee: GDM Holding LLC
G06N3/08
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,499,359
App. No.
17/046,963
Granted
Dec 16, 2025
Kind
B2
Abstract

A graph neural network system for processing data representing a physical system is disclosed. The graph neural network system receives physical system data and action data, where the physical system includes bodies coupled by joints. The graph neural network system processes an input graph to provide an output graph, the input and output graphs each having a graph structure including nodes and edges corresponding, respectively, to the bodies and joints of the physical system. The graph processing neural network provides the inferred static property or dynamic state of the physical system. The graph processing neural network is differentiable and may be used for control and/or reinforcement learning.

Claims (55)

1 . A neural network system for processing data representing a robotic system, the neural network system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

receiving, as an input, robotic data characterizing the robotic system and action data, wherein the robotic system comprises bodies coupled by joints, wherein the robotic data comprises (i) dynamic data representing a dynamic state defined by motions of at least one of the bodies of the robotic system and (ii) static data representing static properties of at least one of the bodies or joints of the robotic system, and wherein the action data represents one or more actions applied to the joints of the robotic system;

processing an input graph derived from the robotic data using a graph processing neural network comprising two or more graph network blocks including (i) a first graph network block trained to process the input graph to provide a latent graph comprising a latent representation of the robotic system; and (ii) a second graph network block trained to process data from the latent graph to provide an output graph,

wherein the input and output graphs each have a graph structure comprising a plurality of nodes connected by a plurality of edges, wherein the nodes represent the bodies of the robotic system and the edges represent the joints of the robotic system,

wherein the input graph has input graph nodes comprising input graph node features and has input graph edges, each input graph edge comprising respective input graph edge features representing one or more actions applied to the respective joint represented by the respective input graph edge, wherein the input graph comprises a dynamic graph and a static graph, the dynamic graph comprising input graph node features representing the dynamic data and input graph edge features representing the action data, the static graph comprising input graph node or edge features representing the static properties of the bodies or joints of the robotic system; and

wherein the output graph has (i) output graph nodes comprising output graph node features and (ii) output graph edges comprising output graph edge features, wherein the output graph node features comprise features for inferring a dynamic state of the robotic system predicting one or more future states of the robotic system;

wherein processing the input graph comprises:

for each of the edges, processing the respective input graph edge features representing the one or more actions applied to the respective joint using an edge neural network to determine the respective output graph edge features;

for each of the nodes, aggregating the output graph edge features for a respective set of edges representing a respective set of joints connecting to the respective body represented by the respective node in the graph structure to determine a set of aggregated edge features for the respective node; and

for each of the nodes, processing the respective aggregated edge features and the respective input graph node features using a node neural network to determine the respective output graph node features; and

providing, as an output, the inferred dynamic state of the robotic system predicting one or more future states of the robotic system, to a controller to control actions to be applied to the robotic system.

2 . The neural network system as claimed in claim 1 wherein processing the input graph edge features using the edge neural network to determine the output graph edge features comprises, for each edge, providing the input graph edge features and the input graph node features for the nodes connected by the edge in the graph structure to the edge neural network to determine the output graph edge features.

3 . The neural network system as claimed in claim 1 wherein the output graph further comprises a global feature output representing a collective state of the output graph edge features and the output graph node features, and wherein the graph processing neural network further comprises a global feature neural network to determine the global feature output.

4 . The neural network system as claimed in claim 3 , wherein the operations further comprise aggregating the set of aggregated edge features for each node and the output graph node features to provide an aggregated graph feature input to the global feature neural network, and wherein the global feature neural network is configured to process the aggregated graph feature input to determine the global feature output.

5 . The neural network system as claimed in claim 1 , wherein the operations further comprise combining the input graph with the latent graph; and wherein the second graph network block is configured to process a combination of the input graph and the latent graph to provide the output graph.

6 . The neural network system as claimed in claim 1 , wherein the input graph comprises a combination of the dynamic graph, the static graph, and a hidden graph;

wherein the graph processing neural network comprises a recurrent graph processing neural network to process the input graph to provide a first layer output graph comprising a combination of the latent graph and an updated hidden graph; and

wherein the operations further comprise providing the latent graph to the second graph network block and to provide the updated hidden graph back to an input of the recurrent graph processing neural network.

7 . The neural network system as claimed in claim 1 , wherein the input is configured to receive dynamic data and action data for a sequence of time steps for defining a sequence of input graphs, wherein, for each of the time steps, the input graph comprises a combination of a dynamic graph and a hidden graph, to define the sequence of input graphs, wherein the dynamic graph comprises the input graph node features representing the dynamic data for the time step and the input graph edge features representing the action data for the time step, wherein the graph processing neural network is configured to process the sequence of input graphs to determine, for each time step, a combination of the output graph and an updated hidden graph, wherein the updated hidden graph provides the hidden graph for the next time step; and wherein, after the sequence of time steps, the output graph comprises a system identification graph in which the output graph node features comprise a representation of static properties of the bodies and/or the joints of the robotic system.

8 . The neural network system as claimed in claim 7 , wherein the operations further comprise concatenating the dynamic graph and the hidden graph to provide the input graph, and splitting the combination of the output graph and the updated hidden graph to update the hidden graph.

9 . The neural network system as claimed in claim 7 , wherein the operations further comprise:

at least one further graph network block configured to receive a combination of the system identification graph and a dynamic graph, wherein the dynamic graph comprises graph node features representing dynamic data for an observed time and graph edge features representing action data for the observed time,

wherein the at least one further graph network block is trained to process the combination of the system identification graph and the dynamic graph to provide a dynamic state prediction graph having node features representing a future dynamic state of the robotic system at a time later than the observed time.

10 . The neural network system as claimed in claim 9 , the operations further comprising using a control system trained to provide action control outputs to maximize a reward predicted from the future dynamic state of the robotic system.

11 . One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for processing data representing a robotic system, the operations comprising:

receiving, as an input, physical system robotic data characterizing the robotic system and action data, wherein the robotic system comprises bodies coupled by joints, wherein the robotic system data comprises at least (i) dynamic data representing a dynamic state defined by motions of at least one of the bodies of the robotic system and (ii) static data representing static properties of at least one of the bodies or joints of the robotic system, and wherein the action data represents one or more actions applied to the joints of the robotic system;

processing an input graph derived from the robotic data using a graph processing neural network comprising two or more graph network blocks including (i) a first graph network block trained to process the input graph to provide a latent graph comprising a latent representation of the robotic system; and (ii) a second graph network block trained to process data from the latent graph to provide an output graph,

wherein the input and output graphs each have a graph structure comprising a plurality of nodes connected by a plurality of edges, wherein the nodes represent the bodies of the robotic system and the edges represent the joints of the robotic system,

wherein the input graph has input graph nodes comprising input graph node features and has input graph edges, each input graph edge comprising respective input graph edge features representing one or more actions applied to the respective joint represented by the respective input graph edge, wherein the input graph comprises a dynamic graph and a static graph, the dynamic graph comprising input graph node features representing the dynamic data and input graph edge features representing the action data, the static graph comprising input graph node or edge features representing the static properties of the bodies or joints of the robotic system; and

wherein the output graph has (i) output graph nodes comprising output graph node features and (ii) output graph edges comprising output graph edge features, wherein the output graph node features comprise features for inferring a dynamic state of the robotic system predicting one or more future states of the robotic system;

wherein processing the input graph comprises:

for each of the edges, processing the respective input graph edge features representing the one or more actions applied to the respective joint using an edge neural network to determine the respective output graph edge features;

for each of the nodes, aggregating the output graph edge features for a respective set of edges representing a respective set of joints connecting to the respective body represented by the respective node in the graph structure to determine a set of aggregated edge features for the respective node; and

for each of the nodes, processing the respective aggregated edge features and the respective input graph node features using a node neural network to determine the respective output graph node features; and

providing, as an output, the inferred dynamic state of the robotic system predicting one or more future states of the robotic system, to a controller to control actions to be applied to the robotic system.

12 . A method performed by one or more computers, the method comprising:

receiving, as an input, robotic data characterizing a robotic system and action data, wherein the robotic system comprises bodies coupled by joints, wherein the robotic data comprises (i) dynamic data representing a dynamic state defined by motions of at least one of the bodies of the robotic system and (ii) static data representing static properties of at least one of the bodies or joints of the robotic system, and wherein the action data represents one or more actions applied to the joints of the robotic system;

processing an input graph derived from the robotic data using a graph processing neural network comprising two or more graph network blocks including (i) a first graph network block trained to process the input graph to provide a latent graph comprising a latent representation of the robotic system; and (ii) a second graph network block trained to process data from the latent graph to provide an output graph,

wherein the input and output graphs each have a graph structure comprising a plurality of nodes connected by a plurality of edges, wherein the nodes represent the bodies of the robotic system and the edges represent the joints of the robotic system,

wherein the input graph has input graph nodes comprising input graph node features and has input graph edges, each input graph edge comprising respective input graph edge features representing one or more actions applied to the respective joint represented by the respective input graph edge, wherein the input graph comprises a dynamic graph and a static graph, the dynamic graph comprising input graph node features representing the dynamic data and input graph edge features representing the action data, the static graph comprising input graph node or edge features representing the static properties of the bodies or joints of the robotic system; and

wherein the output graph has (i) output graph nodes comprising output graph node features and (ii) output graph edges comprising output graph edge features, wherein the output graph node features comprise features for inferring a dynamic state of the robotic system predicting one or more future states of the robotic system;

wherein processing the input graph comprises:

for each of the edges, processing the respective input graph edge features representing the one or more actions applied to the respective joint using an edge neural network to determine the respective output graph edge features;

for each of the nodes, aggregating the output graph edge features for a respective set of edges representing a respective set of joints connecting to the respective body represented by the respective node in the graph structure to determine a set of aggregated edge features for the respective node; and

for each of the nodes, processing the respective aggregated edge features and the respective input graph node features using a node neural network to determine the respective output graph node features; and

providing, as an output, the inferred dynamic state of the robotic system predicting one or more future states of the robotic system, to a controller to control actions to be applied to the robotic system.

13 . The method of claim 12 , wherein processing the input graph edge features using the edge neural network to determine the output graph edge features comprises, for each edge, providing the input graph edge features and the input graph node features for the nodes connected by the edge in the graph structure to the edge neural network to determine the output graph edge features.

14 . The method of claim 12 , wherein the output graph further comprises a global feature output representing a collective state of the output graph edge features and the output

graph node features, and wherein the graph processing neural network further comprises a global feature neural network to determine the global feature output.

15 . The method of claim 14 , wherein the graph processing neural network is further configured to aggregate the set of aggregated edge features for each node and the output graph node features to provide an aggregated graph feature input to the global feature neural network, and wherein the global feature neural network is configured to process the aggregated graph feature input to determine the global feature output.

16 . The method of claim 12 , wherein the method further comprise:

combining the input graph with the latent graph, wherein the second graph network block is configured to process a combination of the input graph and the latent graph to provide the output graph.

17 . The method of claim 12 , wherein the input graph comprises a combination of the dynamic graph, the static graph, and a hidden graph;

wherein the graph processing neural network comprises a recurrent graph processing neural network to process the input graph to provide a first layer output graph comprising a combination of the latent graph and an updated hidden graph; and

wherein the method further comprise providing the latent graph to the second graph network block and to provide the updated hidden graph back to an input of the recurrent graph processing neural network.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 6, 2025
From: DEEPMIND TECHNOLOGIES LIMITED
To: GDM HOLDING LLC
Reel/Frame 071498/0210 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 1, 2021
From: RIEDMILLER, MARTIN; HADSELL, RAIA THAIS; BATTAGLIA, PETER WILLIAM; MEREL, JOSHUA; SPRINGENBERG, JOST; SANCHEZ, ALVARO; HEESS, NICOLAS MANFRED OTTO
To: DEEPMIND TECHNOLOGIES LIMITED
Reel/Frame 055100/0531 →
Continuity (2)
Provisional Application 62656904 · Apr 12, 2018
Related Publication 20210049467A1 · Feb 18, 2021
References Cited (68)
US 11323334B2 · Altshuler · 2022 [cited by examiner]
US 11604957B1 · Schubert · 2023 [cited by examiner]
US 11886352B2 · Liu · 2024 [cited by examiner]
US 20030110300A1 · Chen · 2003 [cited by applicant]
US 20130159231A1 · Modha · 2013 [cited by examiner]
US 20180070864A1 · Schuster · 2018 [cited by examiner]
US 20180204108A1 · Mital · 2018 [cited by examiner]
US 20190005384A1 · Sundar · 2019 [cited by examiner]
US 20200074246A1 · Goyal · 2020 [cited by examiner]
CN 105637540 · 2016 [cited by applicant]
Wang et al. (“NerveNet: Learning Structured Policy with Graph Neural Networks”, ICLR Feb. 15, 2018) (Year: 2018). [cited by examiner]
Gao et al. (“Efficient representation of quantum many-body states with deep neural networks”, nature communication, Sep. 22, 2017) (Year: 2017). [cited by examiner]
Gu et al. (“Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates”, IEEE, May 29-Jun. 3, 2017) (Year: 2017). [cited by examiner]
Kipf et al. (“Neural Relational Inference for Interacting Systems”, Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden, PMLR 80, 2018) (Year: 2018). [cited by examiner]
Amos et al., “Learning awareness models,” CoRR, Apr. 2018, arxiv.org/abs/1804.06318, 19 pages. [cited by applicant]
Atkeson et al., “A comparison of direct and model-based reinforcement learning,” International Conference on Robotics and Automation, 1997, 4:3557-3564. [cited by applicant]
Battaglia et al., “Interaction Networks for Learning about Objects, Relations and Physics,” Advances in Neural Information Processing Systems 29, 2016, 12 pages. [cited by applicant]
Battaglia et al., “Simulation as an engine of physical scene understanding,” Proceedings of the National Academy of Sciences, 2013, 110(45):18327-18332. [cited by applicant]
Bronstein et al., “Geometric deep learning: going beyond euclidean data,” IEEE Signal Processing Magazine, 2017, 34(4):18-42. [cited by applicant]
Bruna et al., “Spectral networks and locally connected networks on graphs,” CoRR, Dec. 2013, arXiv:1312.6203, 14 pages. [cited by applicant]
Chang et al., “A compositional object-based approach to learning physical dynamics,” CoRR, Dec. 2016, arXiv:1612.00341, 15 pages. [cited by applicant]
Cho et al., “On the properties of neural machine translation: Encoder-decoder approaches,” CoRR, Sep. 2014, arXiv:1409.1259, 9 pages. [cited by applicant]
Defferrard et al., “Convolutional neural networks on graphs with fast localized spectral filtering,” Advances in Neural Information Processing Systems 29, 2016, 9 pages. [cited by applicant]
Deisenroth et al., “Pilco: A model-based and data-efficient approach to policy search,” Proceedings of the 28th International Conference on Machine Learning, 2011, 8 pages. [cited by applicant]
Duvenaud et al., “Convolutional networks on graphs for learning molecular fingerprints,” Advances in Neural Information Processing Systems 28, 2015, 9 pages. [cited by applicant]
Ehrhardt et al., “Learning a physical long-term predictor,” CoRR, Mar. 2017, arXiv:1703.00247, 10 pages. [cited by applicant]
Fragkiadaki et al., “Learning visual predictive models of physics for playing billiards,” CoRR, Nov. 2015, arxiv.org/abs/1511.07404, 12 pages. [cited by applicant]
Gilmer et al., “Neural message passing for quantum chemistry,” CoRR, Apr. 2017, arXiv:1704.01212, 14 pages. [cited by applicant]
Grzeszczuk et al., “Neuroanimator: Fast neural network emulation and control of physics-based models,” Proceedings of the 25th annual conference on Computer graphics and interactive techniques, Jul. 1998, pp. 9-20. [cited by applicant]
Gu et al., “Continuous deep q-learning with modelbased acceleration,” Proceedings of the 33rd International Conference on Machine Learning, 2016, 48: 10 pages. [cited by applicant]
Hamrick et al., “Metacontrol for adaptive imagination-based optimization,” CoRR, May 2017, arXiv:1705.02670, 21 pages. [cited by applicant]
Heess et al., “Learning Continuous Control Policies by Stochastic Value Gradients,” Advances in Neural Information Processing Systems 28, 2015, 9 pages. [cited by applicant]
Henaff et al., “Deep convolutional networks on graph-structured data,” CoRR, Jun. 2015, arXiv:1506.05163, 10 pages. [cited by applicant]
Houthooft et al., “VIME: Variational Information Maximizing Exploration,” CoRR, May 2016, arxiv.org/abs/1605.09674, 11 pages. [cited by applicant]
Johnson-Laird “Mental models in cognitive science,” Cognitive Science, 1980, 4(1):71-115. [cited by applicant]
Kingma et al., “Auto-Encoding Variational Bayes,” CoRR, Dec. 2013, arXiv:1312.6114, 14 pages. [cited by applicant]
Kipf et al., “Semi-supervised classification with graph convolutional networks,” CoRR, Sep. 2016, arXiv:1609.02907, 14 pages. [cited by applicant]
Levine et al., “Learning neural network policies with guided policy search under unknown dynamics,” Advances in Neural Information Processing Systems 27, 2014, 9 pages. [cited by applicant]
Li et al., “Gated graph sequence neural networks,” CoRR, Nov. 2015, arXiv:1511.05493, 20 pages. [cited by applicant]
Li et al., “Iterative linear quadratic regulator design for nonlinear biological movement systems,” In Proceedings of 1st International Conference on Informatics in Control, Automation and Robotics, 2004, 8 pages. [cited by applicant]
Lillicrap et al., “Continuous control with deep reinforcement learning,” CoRR, Sep. 2015, arxiv.org/abs/1509.02971, 14 pages. [cited by applicant]
Miall et al., “Forward models for physiological motor control,” Neural Networks, 1996, 9(8):1265-1279. [cited by applicant]
Nagabandi et al., “Neural network dynamics for modelbased deep reinforcement learning with model-free finetuning,” 2018 IEEE International Conference on Robotics and Automation, May 2018, 10 pages. [cited by applicant]
Niepert et al., “Learning convolutional neural networks for graphs,” Proceedings of the 33 rd International Conference on Machine Learning, 2016, 48:2014-2023. [cited by applicant]
Pascanu et al., “Learning model-based planning from scratch,” CoRR, Jul. 2017, arxiv.org/abs/1707.06170, 13 pages. [cited by applicant]
PCT International Preliminary Report on Patentability in International International Appln. No. PCT/EP2019/059431, dated Oct. 13, 2020, 13 pages. [cited by applicant]
PCT International Search Report and Written Opinion in International Appln. No. PCT/EP2019/059431, dated Jul. 25, 2019, 19 pages. [cited by applicant]
Peng et al., “Sim-to-real transfer of robotic control with dynamics randomization,” 2018 IEEE International Conference on Robotics and Automation, May 2018, 8 pages. [cited by applicant]
Rajeswaran et al., “EPOpt: Learning robust neural network policies using model ensembles,” CoRR, Oct. 2016, arxiv.org/abs/1610.01283, 15 pages. [cited by applicant]
Raposo et al., “Discovering objects and their relations from entangled scene representations,” CoRR, Feb. 2017, arXiv:1702.05068, 16 pages. [cited by applicant]
Rezende et al., “Stochastic backpropagation and approximate inference in deep generative models,” CoRR, Jan. 2014, arxiv.org/abs/1401.4082, 14 pages. [cited by applicant]
Sanchez-Gonzalez et al., “Graph Networks as Learnable Physics Engines for Inference and Control,” CoRR, arxiv.org/abs/1806.01242, 21 pages. [cited by applicant]
Santoro et al., “A simple neural network module for relational reasoning,” Advances in Neural Information Processing Systems 30, 2017, 10 pages. [cited by applicant]
Scarselli et al., “Computational capabilities of graph neural networks,” IEEE Transactions on Neural Networks, 2009, 20(1):81-102. [cited by applicant]
Scarselli et al., “Graph neural networks for ranking web pages,” The 2005 IEEE/WIC/ACM International Conference on Web Intelligence, Sep. 2005, 7 pages. [cited by applicant]
Scarselli et al., “The graph neural network model,” IEEE Transactions on Neural Networks, 2009, 20(1):61-80. [cited by applicant]
Schmidhuber et al., “Curious model-building control systems,” Proc. Int. J. Conf. Neural Networks, 1991, 7 pages. [cited by applicant]
Spelke et al., “Core knowledge,” Developmental science, Jan. 2007, 10(1):89-96. [cited by applicant]
Sun et al., “Planning to be surprised: Optimal bayesian exploration in dynamic environments,” Artificial General Intelligence, Lecture Notes in Computer Science, 2011, 6830:41-51. [cited by applicant]
Tassa et al., “Control-limited differential dynamic programming,” 2014 IEEE International Conference on Robotics and Automation, May 2014, 8 pages. [cited by applicant]
Tassa et al., “Deepmind control suite,” CoRR, Jan. 2018, arXiv:1801.00690, 24 pages. [cited by applicant]
Tassa et al., “Receding horizon differential dynamic programming,” Advances in Neural Information Processing Systems 20, 2007, pp. 1465-1472. [cited by applicant]
Todorov et al., “Mujoco: A physics engine for model-based control,” 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Oct. 2012, pp. 5026-5033. [cited by applicant]
Wang et al., “Nervenet: Learning Structured Policy with Graph Neural Networks,” 6th International Conference on Learning Representations, Mar. 2018, pp. 1-26. [cited by applicant]
Watters et al., “Visual Interaction Networks,” Advances in Neural Information Processing Systems 30, 2017, 14 pages. [cited by applicant]
Yu et al., “Preparing for the unknown: Learning a universal policy with online system identification,” CoRR, Feb. 2017, arxiv.org/abs/1702.02453, 10 pages. [cited by applicant]
Office Action in European Appln. No. 19718138.1, Feb. 6, 2023, 11 pages. [cited by applicant]
Office Action in Chinese Appln. No. 201980025463.4, Aug. 31, 2023, 33 pages (with English translation). [cited by applicant]