IP Library › Granted Patent US 12,731,317
Granted Patent B2
US 12,731,317 · App. 17/842,481 · Granted Sep 8, 2026

Physics-based image generation using one or more neural networks

Inventors: Tingwu Wang (Toronto, CA); Yunrong Guo (Richmond Hill, CA); Cheng Xie (Vancouver, CA); Xue Bin Peng (Vancouver, CA); Sanja Fidler (Toronto, CA)
Assignee: NVIDIA Corporation
G06T13/40G06N3/04G06T17/10
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,731,317
App. No.
17/842,481
Granted
Sep 8, 2026
Kind
B2
Abstract

Apparatuses, systems, and techniques are presented to generate images representing realistic motion or activity. In at least one embodiment, one or more neural networks are used to generate one or more images of one or more characters performing one or more actions based, at least in part, upon one or more physical capabilities of the one or more characters.

Claims (51)

1 . A processor, comprising:

one or more circuits to:

cause one or more neural networks to generate one or more actions of one or more characters based, at least in part, on:

propagating respective information between adjacent nodes of a character graph to update respective information of the adjacent nodes of a given character graph of the one or more characters, wherein the respective information of an individual node of the adjacent nodes for a given character graph of the one or more characters indicates:

one or more physical capabilities for the one or more characters;

a current state for the individual node for the given character graph; and

a target state for the individual node for the given character graph; and;

cause a rendering of one or more images depicting a performance of the one or more actions based, at least in part, on the updated respective information of the adjacent nodes of the given character graph, wherein the one or more actions depicted in the one or more images are performed according to the one or more physical capabilities of the one or more characters to perform the one or more actions.

2 . The processor of claim 1 , wherein the given character graph includes a connected set of nodes, including the adjacent nodes, representing body components of the one or more characters having corresponding parameter values for the one or more physical capabilities.

3 . The processor of claim 2 , wherein the one or more circuits are further to determine motions for the connected set of nodes of the character graph based, at least in part, upon the current state for the individual node and target state for the individual node.

4 . The processor of claim 3 , wherein the one or more comprise one or more graph neural networks that propagate the respective information by exchanging state information between the connected set of the nodes for each of a set of forward passes through the one or more graph neural networks.

5 . The processor of claim 2 , wherein the one or more circuits are further to generate different character graphs for different characters having different morphologies or bodily structures.

6 . The processor of claim 1 , wherein the one or more circuits are to cause the one or more images to be rendered in a sequence to produce physics-based animation for the one or more characters.

7 . A system comprising one or more processors to:

cause one or more neural networks to one or more actions of one or more characters based, at least in part, on:

propagating respective information between adjacent nodes of a character graph to update respective information of the adjacent nodes of a given character graph of the one or more characters, wherein the respective information of an individual node of the adjacent nodes for a given character graph of the one or more characters indicates:

one or more physical capabilities for the one or more characters;

a current state for the individual node for the given character graph; and

a target state for the individual node for the given character graph; and;

cause a rendering of one or more images depicting a performance of the one or more actions based, at least in part, on the updated respective information of the adjacent nodes of the given character graph, wherein the one or more actions depicted in the one or more images are performed according to the one or more physical capabilities of the one or more characters to perform the one or more actions.

8 . The system of claim 7 , wherein the given character graph includes a connected set of nodes, including the adjacent nodes, representing body components of the one or more characters having corresponding parameter values for the one or more physical capabilities.

9 . The system of claim 8 , wherein the one or more processors are further to determine motions for the connected set of nodes of the character graph based, at least in part, upon the current state for the individual node and target state for the individual node.

10 . The system of claim 9 , wherein the one or more neural networks comprise one or more graph neural networks to propagate the respective information between the adjacent nodes by exchanging state information between the connected set of the nodes for each of a set of forward passes through the one or more graph neural networks.

11 . The system of claim 8 , wherein the one or more processors are further to generate different character graphs for different characters having different morphologies or bodily structures.

12 . The system of claim 7 , wherein the one or more images are rendered in a sequence to produce physics-based animation for the one or more characters.

13 . A method comprising:

causing one or more neural networks to generate one or more actions of one or more characters based, at least in part, on:

propagating respective information between adjacent one or more nodes of a character graph to update respective information of the adjacent nodes of a given character graph of the one or more characters, wherein the respective information of an individual node of the adjacent nodes for a given character graph of the one or more characters represents information indicates:

one or more physical capabilities for the one or more characters;

a current state for the individual node for the given character graph; and

a target state for the individual node for the given character graph; and;

causing a rendering of one or more images depicting a performance of the one or more actions based, at least in part, on the updated respective information of the adjacent nodes of the given character graph, wherein the one or more actions depicted in the one or more images are performed according to the one or more physical capabilities of the one or more characters to perform the one or more actions.

14 . The method of claim 13 , wherein the given character graph includes a connected set of nodes, including the adjacent nodes, representing body components of the one or more characters having corresponding parameter values for the one or more physical capabilities.

15 . The method of claim 14 , further comprising:

determining motions for the connected set of nodes of the character graph based, at least in part, upon the current state for the individual node and target state for the individual node.

16 . The method of claim 15 , wherein the one or more neural networks comprise one or more graph neural networks to propagate the respective information between the adjacent nodes by exchanging state information between the connected set of the nodes for each of a set of forward passes through the one or more graph neural networks.

17 . The method of claim 14 , further comprising:

generating different character graphs for different characters having different morphologies or bodily structures.

18 . The method of claim 13 , wherein the one or more images are rendered in a sequence to produce physics-based animation for the one or more characters.

19 . An image generation system, comprising:

one or more processors to cause one or more neural networks to generate one or more actions of one or more characters based, at least in part, on:

propagating respective information between adjacent nodes of a character graph to update respective information of the adjacent nodes of a given character graph of the one or more characters, wherein the respective information of an individual node of the adjacent nodes for a given character graph of the one or more characters indicates:

one or more physical capabilities for the one or more characters;

a current state for the individual node for the given character graph; and

a target state for the individual node for the given character graph; and;

cause a rendering of one or more images depicting a performance of the one or more actions based, at least in part, on the updated respective information of the adjacent nodes of the given character graph, wherein the one or more actions depicted in the one or more images are performed according to the one or more physical capabilities of the one or more characters to perform the one or more actions.

20 . The image generation system of claim 19 , wherein the given character graph includes a connected set of nodes, including the adjacent nodes, representing body components of the one or more characters having corresponding parameter values for the one or more physical capabilities.

21 . The image generation system of claim 20 , wherein the one or more processors are further to determine motions for the connected set of nodes of the character graph based, at least in part, upon the current state for the individual node and target state for the individual node.

22 . The image generation system of claim 21 , wherein the one or more neural networks comprise one or more graph neural networks to propagate the respective information between the adjacent nodes by exchanging state information between the connected set of the nodes for each of a set of forward passes through the one or more graph neural networks.

23 . The image generation system of claim 22 , wherein the one or more processors are further to generate different character graphs for different characters having different morphologies or bodily structures.

24 . The image generation system of claim 19 , wherein the one or more processors are further to generate the one or more images in a sequence to produce physics-based animation for the one or more characters.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 24, 2022
From: WANG, TINGWU; GUO, YUNRONG; XIE, CHENG; PENG, XUE BIN; FIDLER, SANJA
To: NVIDIA CORPORATION
Reel/Frame 060310/0079 →
Continuity (1)
Related Publication 20230410397A1 · Dec 21, 2023
References Cited (90)
US 11928764B2 · Wang et al. · 2024 [cited by applicant]
US 20190134915A1 · Schmidt · 2019 [cited by examiner]
US 20200372658A1 · De Winde · 2020 [cited by examiner]
US 20220386942A1 · Diaz-Arias · 2022 [cited by examiner]
US 20230123820A1 · Wang · 2023 [cited by examiner]
US 20230267317A1 · Shin · 2023 [cited by examiner]
Aberman et al., “Learning Character-Agnostic Motion for Motion Retargeting in 2D,” May 5, 2019, 14 pages. [cited by applicant]
Aberman et al., “Skeleton-Aware Networks for Deep Motion Retargeting,” May 12, 2020, 14 pages. [cited by applicant]
Andrychowicz et al., “Learning Dexterous In-Hand Manipulation,” Aug. 28, 2018, 27 pages. [cited by applicant]
Arikan et al., “Interactive Motion Generation from Examples,” ACM Transactions on Graphics, 21(3): 2002, 8 pages. [cited by applicant]
Bergamin et al., “DReCon: Data-Driven Responsive Control of Physics-Based Characters,” ACM Transactions on Graphics, 38(6): 2019, 11 pages. [cited by applicant]
Bruna et al., “Spectral networks and locally connected networks on graphs,” ICLR, 2014, 14 pages. [cited by applicant]
Buttner, “Machine Learning for Motion Synthesis and Character Control,” In Interactive 3D Graphics and Games, retrieved from https://www.youtube.com/watch?v=zuvmQxcCOM4, Jul. 9, 2019, 3 pages. [cited by applicant]
Buttner, “Motion Matching—The Road to Next-Gen Animation,” In Nucl. ai Conference, retrieved from https://www.youtube.com/watch?v=z_wpgHFSWss, 2015, 3 pages. [cited by applicant]
Chentanez et al., “Physics-based Motion Capture Imitation with Deep Reinforcement Learning,” In Proceedings of the 11th Annual International Conference on Motion,Interaction, and Games, ACM 1, 2018, 10 pages. [cited by applicant]
Clavet, “Motion Matching and the Road to Next-Gen Animation,” Process of Game Developers Conference, Mar. 14-18, 2016, 107 pages. [cited by applicant]
Defferrard et al., “Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering,” NIPS, 2016, 9 pages. [cited by applicant]
Duan et al., “Benchmarking Deep Reinforcement Learning for Continuous Control,” May 27, 2016, 14 pages. [cited by applicant]
Duvenaud et al., “Convolutional networks on graphs for learning molecular fingerprints,” NIPS, 2015, 9 pages. [cited by applicant]
Erleben et al., “Physics-Based Animation,” Charles River Media Hingham, 2005, 830 pages. [cited by applicant]
Gilmer et al., “Neural Message Passing for Quantum Chemistry,” Jun. 12, 2017, 14 pages. [cited by applicant]
Gori et al., “A New Model for Learning in Graph Domains,” IEEE International Joint Conference on Neural Networks, 2005, 6 pages. [cited by applicant]
Gu et al., “Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates,” ICRA, 2017, 9 pages. [cited by applicant]
Gupta et al., “Embodied Intelligence via Learning and Evolution,” Feb. 3, 2021, 18 pages. [cited by applicant]
Heess et al., “Emergence of Locomotion Behaviours in Rich Environments,” Jul. 10, 2017, 14 pages. [cited by applicant]
Ho et al., “Generative Adversarial Imitation Learning,” NIPS, 2016, 9 pages. [cited by applicant]
Holden et al., “Learned motion matching,” ACM Transactions on Graphics, 39(4): 2020, 13 pages. [cited by applicant]
Holden et al., “Phase-Functioned Neural Networks for Character Control,” ACM Transactions on Graphics, 36(4): Jul. 2017, 13 pages. [cited by applicant]
Huang et al., “One policy to control them all: Shared modular policies for agent-agnostic control,” International Conference on Machine Learning, 2020, 10 pages. [cited by applicant]
Hämäläinen et al., “Online Control of Simulated Humanoids using Particle Belief Propagation,” ACM Transactions on Graphics, 34(4): 2015, 13 pages. [cited by applicant]
IEEE “IEEE Standard for Floating-Point Arithmetric”, Microprocessor Standards Committee of the IEEE Computer Society, IEEE Std 754-2008, dated Jun. 12, 2008, 70 pages. [cited by applicant]
Jain et al., “Structural-RNN: Deep Learning on Spatio-Temporal Graphs,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, 10 pages. [cited by applicant]
James et al., “Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, 11 pa… [cited by applicant]
Kaiser et al., “Model-Based Reinforcement Learning for Atari,” Jun. 11, 2019, 18 pages. [cited by applicant]
Kajita et al., “Biped Walking Pattern Generation by a Simple Three-Dimensional Inverted Pendulum Model,” Advanced Robotics, 17(2): 2003, 18 pages. [cited by applicant]
Kipf et al., Semi-supervised Classification with Graph Convolutional Networks, Nov. 3, 2016, 13 pages. [cited by applicant]
Kovar “Motion Graphs,” Proceedings of SIGGRAPH, 2002, 10 pages. [cited by applicant]
Kovar et al., “Automated Extraction and Parameterization of Motions in Large Data Sets,” ACM Transactions on Graphics, 23(3): 2004, 10 pages. [cited by applicant]
Kwon et al., “Fast and Flexible Multilegged Locomotion Using Learned Centroidal Dynamics,” ACM Transactions on Graphics, 39(4): 2020, 24 pages. [cited by applicant]
Lamprecht et al., “Renderpeople,” Renderpeople, retrieved from https://renderpeople.com/, 2021, 9 pages. [cited by applicant]
Lee et al., “Interactive Control of Avatars Animated with Human Motion Data,” Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques, 2002, 10 pages. [cited by applicant]
Lee et al., “Learning a family of motor skills from a single motion clip,” ACM Transactions on Graphics, 40(4): 2021, 13 pages. [cited by applicant]
Lee et al., “Motion Fields for Interactive Character Locomotion,” ACM SIGGRAPH Asia 2010 Papers Association for Computing Machinery, 2010, 8 pages. [cited by applicant]
Li et al., “Gated Graph Sequence Neural Networks,” Nov. 19, 2015, 19 pages. [cited by applicant]
Liang et al., “CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving,” ECCV, 2018, 16 pages. [cited by applicant]
Lillicrap et al., “Continuous Control with Deep Reinforcement Learning,” Nov. 18, 2015, 14 pages. [cited by applicant]
Liu et al., “Guided Learning of Control Graphs for Physics-based Characters,” ACM Transactions on Graphics, 35(3): 2016, 14 pages. [cited by applicant]
Liu et al., “Improving Sampling-based Motion Control,” In Computer Graphics Forum, vol. 34, Wiley Online Library, 2015, 9 pages. [cited by applicant]
Liu et al., “Sampling-based Contact-rich Motion Control,” ACM Transactions on Graphics, 29(4): 2010, 10 pages. [cited by applicant]
Makoviychuk et al., “Isaac Gym: High Performance GPU-Based Physics Simulation for Robot Learning,” Aug. 25, 2021, 32 pages. [cited by applicant]
Merel et al., “Learning Human Behaviors from Motion Capture by Adversarial Imitation,” Jul. 10, 2017, 12 pages. [cited by applicant]
Min et al., “Motion Graphs++ A Compact Generative Model for Semantic Motion Analysis and Synthesis,” ACM Transactions on Graphics, 31(6): 2012, 12 pages. [cited by applicant]
Mnih et al., “Playing Atari with Deep Reinforcement Learning,” Dec. 19, 2013, 9 pages. [cited by applicant]
Mordatch et al., “Discovery of complex behaviors through contact-invariant optimization,” ACM Transactions on Graphics, 31(4): 2012, 8 pages. [cited by applicant]
Mordatch et al., “Robust Physics-Based Locomotion Using Low-Dimensional Planning,” ACM SIGGRAPH, 2010, 8 pages. [cited by applicant]
Park et al., “Learning Predict-and-Simulate Policies from Unorganized Human Motion Data,” ACM Transactions on Graphics, 38(6): 2019, 11 pages. [cited by applicant]
Pathak et al., “Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity,” Nov. 21, 2019, 14 pages. [cited by applicant]
Peng et al. “Deepmimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills”, Jul. 27, 2018, 18 pages. [cited by applicant]
Peng et al., “AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control,” ACM Transactions on Graphics., 40(4): Jul. 2021, 20 pages. [cited by applicant]
Peng et al., “MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies,” May 23, 2019, 18 pages. [cited by applicant]
Peng et al., “SFV: Reinforcement Learning of Physical Skills from Videos,” ACM Transactions on Graphics, 37(6): 2018, 17 pages. [cited by applicant]
Qi et al., “3D Graph Neural Networks for RGBD Semantic Segmentation,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, 10 pages. [cited by applicant]
Safonova et al., “Construction and Optimal Search of Interpolated Motion Graphs,” ACM SIGGRAPH papers, 2007, 27 pages. [cited by applicant]
Sallab et al., “Deep Reinforcement Learning Framework for Autonomous Driving,” Electronic Imaging, 2017, 7 pages. [cited by applicant]
Sanchez-Gonzalez et al., “Graph Networks as Learnable Physics Engines for Inference and Control,” International Conference on Machine Learning, 2018, 10 pages. [cited by applicant]
Scarselli et al., “The Graph Neural Network Model,” IEEE TNN, 2009, 21 pages. [cited by applicant]
Schlichtkrull et al., “Modeling Relational Data with Graph Convolutional Networks,” European semantic web conference, Oct. 26, 2017, 9 pages. [cited by applicant]
Schulman et al., “Proximal policy optimization algorithms,” arXiv preprint arXiv: 1707.06347, Jul. 20, 2017, 12 pages. [cited by applicant]
Shi et al., “MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistency,” ACM Transactions on Graphics, 40(1): 2020, 15 pages. [cited by applicant]
Silver et al., “Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm,” Dec. 5, 2017, 19 pages. [cited by applicant]
Silver et al., “Mastering the Game of Go with Deep Neural Networks and Tree Search,” Nature, 2016, 37 pages. [cited by applicant]
Socher et al., “Parsing Natural Scenes and Natural Language with Recursive Neural Networks,” Proceedings of the 28th International Conference on Machine Learning, 2011, 8 pages. [cited by applicant]
Society of Automotive Engineers On-Road Automated Vehicle Standards Committee, “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles,” Standard No. J3016-201609, issued Jan… [cited by applicant]
Society of Automotive Engineers On-Road Automated Vehicle Standards Committee, “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles,” Standard No. J3016-201806, issued Jan… [cited by applicant]
Starke et al., “Local Motion Phases for Learning Multi-Contact Character Movements,” ACM Transactions on Graphics, 39(4): 2020, 14 pages. [cited by applicant]
Starke et al., “Neural Animation Layering for Synthesizing Martial Arts Movements,” ACM Transactions on Graphics, 40(4): 2021, 16 pages. [cited by applicant]
Starke et al., “Neural State Machine for Character-Scene Interactions,” ACM Transactions on Graphics, 38(6): 2019, 14 pages. [cited by applicant]
Tai et al., “Virtual-to-real Deep Reinforcement Learning: Continuous Control of Mobile Robots for Mapless Navigation,” IEEE/RSJ International Conference on Intelligent Robots and Systems, 2017, 6 pages. [cited by applicant]
Todorov et al., “MuJoCo : A Physics Engine for Model-Based Control,” 2012, 8 pages. [cited by applicant]
Tsai et al., “Real-time Physics-Based 3D Biped Character Animation Using an Inverted Pendulum Model,” IEEE transactions on visualization and computer graphics, 16(2): 2009, 13 pages. [cited by applicant]
Wang et al., “Benchmarking Model-based Reinforcement Learning,” Jul. 3, 2019, 25 pages. [cited by applicant]
Wang et al., “NerveNet: Learning Structured Policy with Graph Neural Networks,” International Conference on Learning Representations, 2018, 26 pages. [cited by applicant]
Wang et al., “Neural Graph Evolution: Towards Efficient Automatic Robot Design,” Jun. 12, 2019, 17 pages. [cited by applicant]
Wang et al., “UniCon: Universal Neural Controller For Physics-based Character Motion,” Nov. 30, 2020, 15 pages. [cited by applicant]
Won et al., “A scalable approach to control diverse behaviors for physically simulated characters,” ACM Transactions on Graphics, 39(4): 2020, 12 pages. [cited by applicant]
Won et al., “Learning Body Shape Variation in Physics-Based Characters,” ACM Transactions on Graphics, 38(6): 2019, 12 pages. [cited by applicant]
Yao et al., “Graph Convolutional Networks for Text Classification,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, 2019, 8 pages. [cited by applicant]
Yi et al., “Search Space Reduction in Motion Matching by Trajectory Clustering,” SIGGRAPH Asia, 2019, 2 pages. [cited by applicant]
Zhang et al., “Mode-Adaptive Neural Networks for Quadruped Motion Control,” ACM Transactions on Graphics, 37(4): 2018, 11 pages. [cited by applicant]
Zhu et al., “Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning,” IEEE International Conference on Robotics and Automation, 2016, 8pages. [cited by applicant]