IP Library › Granted Patent US 12,198,274
Granted Patent B2
US 12,198,274 · App. 17/618,783 · Granted Jan 14, 2025

Deep learning of biomimetic sensorimotor control for biomechanical model animation

Inventors: Demetri Terzopoulos (Los Angeles, CA); Masaki Nakada (Los Angeles, CA)
Assignee: The Regents of the University of California
G06T19/003A61B3/0025G06F30/20G06T7/0012G06T2207/30041
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,198,274
App. No.
17/618,783
Granted
Jan 14, 2025
Kind
B2
Abstract

Systems and methods for training and implementing a sensorimotor control framework in accordance with embodiments of the invention are illustrated. One embodiment includes a method for operating a sensorimotor control framework. The method includes steps for determining a current state of a musculoskeletal model, determining a desired state of the musculoskeletal model, computing a set of muscle activations, based on a difference between the current and desired states, and driving a musculoskeletal model based on the computed muscle activations.

Claims (39)

1. A method for operating a sensorimotor control framework, the method comprising:

determining a current state of a musculoskeletal model comprising a torso complex and a set of one or more limb complexes attached to the torso complex;

determining a desired state of the musculoskeletal model, wherein the current and desired states comprise at least one of a position, an orientation, a muscle strain, and a strain rate;

computing forces exerted by the set of limb complexes;

computing a set of muscle activations for the torso complex, based on the forces exerted by the set of limb complexes and a difference between the current and desired states of the musculoskeletal model; and

driving a musculoskeletal model based on the computed muscle activations.

2. The method of claim 1 , wherein the musculoskeletal model is a model of a human being and comprises a plurality of Hill-type actuators.

3. The method of claim 1 , wherein the musculoskeletal model comprises a plurality of complexes, computing the set of muscle activations comprises computing a first set of muscle activations for a first complex and computing a second set of muscle activations for a second complex.

4. The method of claim 3 , wherein the plurality of complexes comprises at least one of an arm complex, a leg complex, a cervicocephalic complex, and a torso complex.

5. The method of claim 3 , wherein computing the second set of muscle activations for the second complex comprises:

computing forces from the first set of muscle activations; and

computing the second set of muscle activations based on the computed forces.

6. The method of claim 1 further comprising receiving sensory inputs from a sensory element, wherein the sensory element is an eye model and the sensory inputs comprise a one-dimensional optic nerve vector (ONV).

7. The method of claim 6 , wherein the eye model has a non-uniform distribution of photoreceptors on the retina.

8. The method of claim 1 , wherein computing the set of muscle activations comprises:

computing a voluntary signal using a voluntary controller;

computing a reflex signal using a reflex controller; and

computing the set of muscle activations based on outputs from the voluntary controller and outputs from the reflex controller.

9. The method of claim 8 , wherein the voluntary controller takes as input a target orientation and output activations for muscles of an associated muscle group.

10. The method of claim 8 , wherein the reflex controller takes as input muscle strains and strain rates of muscles of an associated muscle group.

11. A method for training a torso controller of a torso complex, the method comprising:

receiving training data comprising a set of target inputs and a target state of the torso complex and limb complexes attached to the torso complex;

determining forces exerted by the limb complexes attached to the torso complex;

computing muscle activations for the torso complex based on the target inputs, the target state, and the determined forces;

computing differences between a result of the muscle activations and the target state; and

modifying parameters of the torso controller based on the differences between the target state and a current state of the torso complex and the limb complexes, wherein the current and target states comprise at least one of a position, an orientation, a muscle strain, and a strain rate.

12. The method of claim 11 further comprising synthesizing the training data.

13. The method of claim 12 , wherein synthesizing the training data comprises:

specifying a current position and a target position for a model;

determining discrepancies between the current position and the target position;

determining muscle activation adjustments based on the discrepancies and muscle activations of the current position;

determining discrepancies comprises solving an inverse kinematics problem.

14. The method of claim 13 , wherein the discrepancies comprise at least one of an angular discrepancy, a linear discrepancy, and an acceleration discrepancy.

15. The method of claim 13 , wherein determining muscle activation adjustments comprises performing a muscle optimization technique to compute the minimal muscle activations that can generate a desired torque.

16. The method of claim 12 , wherein synthesizing the training data comprises:

specifying current and desired muscle strains and strain rates for a model;

determining changes in muscle strain and in strain rate for muscles of the controlled group; and

determining muscle activation adjustments based on the changes in muscle strain and in strain rate.

17. The method of claim 12 , wherein determining forces exhibited by limb complexes comprises computing random forces for the set of limb complexes by sending activation signals to the limb complexes.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 27, 2023
From: TERZOPOULOS, DEMETRI; NAKADA, MASAKI
To: THE REGENTS OF THE UNIVERSITY OF CALIFORNIA
Reel/Frame 064088/0340 →
Continuity (2)
Provisional Application 62861911 · Jun 14, 2019
Related Publication 20220262075A1 · Aug 18, 2022
References Cited (87)
US 9585631B2 · Vappou · 2017 [cited by examiner]
US 10140745B2 · Pai · 2018 [cited by examiner]
US 11052288B1 · Berme · 2021 [cited by examiner]
US 11301045B1 · Berme · 2022 [cited by examiner]
US 11311209B1 · Berme · 2022 [cited by examiner]
US 11501034B2 · De Vries · 2022 [cited by examiner]
US 20070172797A1 · Hada et al. · 2007 [cited by applicant]
US 20080221487A1 · Zohar · 2008 [cited by examiner]
US 20090082701A1 · Zohar et al. · 2009 [cited by applicant]
US 20160203630A1 · Pai et al. · 2016 [cited by applicant]
US 20180024635A1 · Kaifosh et al. · 2018 [cited by applicant]
US 20180336300A1 · De Vries · 2018 [cited by examiner]
US 20190227627A1 · Kaifosh · 2019 [cited by examiner]
US 20190228590A1 · Kaifosh · 2019 [cited by examiner]
US 20190228591A1 · Giurgica-Tiron · 2019 [cited by examiner]
US 20200069210A1 · Berenzweig · 2020 [cited by examiner]
US 20200118350A1 · Kaifosh · 2020 [cited by examiner]
US 20200125172A1 · Kaifosh · 2020 [cited by examiner]
US 20210405750A1 · Kaifosh · 2021 [cited by examiner]
EP 3984001A1 · 2022 [cited by applicant]
JP 2007213015A · 2007 [cited by applicant]
JP 2022536349A · 2022 [cited by applicant]
WO 2020252484A1 · 2020 [cited by applicant]
Extended European Search Report for European Application No. 20823697.6, Search completed May 3, 2023, Mailed May 12, 2023, 16 Pgs. [cited by applicant]
International Preliminary Report on Patentability for International Application PCT/US2020/037811, Report issued Dec. 14, 2021, Mailed Dec. 23, 2021, 9 Pgs. [cited by applicant]
International Search Report and Written Opinion for International Application No. PCT/US2020/037811, Search completed Aug. 12, 2020, Mailed Aug. 31, 2020, 15 Pgs. [cited by applicant]
Bergstra et al., “Theano: A CPU and GPU Math Compiler in Python”, Proceedings of the 9th Python in Science Conference, Austin, Texas, 2010, 7 pgs. [cited by applicant]
Cruz Ruiz et al., “Muscle-Based Control for Character Animation”, Computer Graphics Forum, vol. 36, No. 6, Sep. 2017, pp. 122-147. [cited by applicant]
Deering, “A Photon Accurate Model of the Human Eye”, ACM Transactions on Graphics, vol. 24, No. 3, Jul. 2005, pp. 649-658. [cited by applicant]
Elgammal et al., “CAN: Creative Adversarial Networks Generating “Art” by Learning About Styles and Deviating from Style Norms”, arXiv:1706.07068v1 [cs.AI], Jun. 21, 2017, 22 pgs. [cited by applicant]
Faloutsos et al., “Composable Controllers for Physics-Based Character Animation”, Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, Los Angeles, California, Aug. 12-17, 2001, 10 … [cited by applicant]
Fan et al., “Active Volumetric Musculoskeletal Systems”, ACM Transactions on Graphics, vol. 33, No. 4, Jul. 2014, Article 152, 9 pgs. [cited by applicant]
Featherstone, “Rigid Body Dynamics Algorithms”, Springer-Verlag Berlin, Heidelberg, 2007, 281 pgs. [cited by applicant]
Ganin et al., “Synthesizing Programs for Images using Reinforced Adversarial Learning”, arXiv:1804.01118v1 [cs.CV], Apr. 3, 2018, 12 pgs. [cited by applicant]
Gatys, et al., “Image Style Transfer Using Convolutional Neural Networks”, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2414-2423. [cited by applicant]
Geijtenbeek et al., “Flexible Muscle-Based Locomotion for Bipedal Creatures”, ACM Transactions on Graphics, vol. 32, No. 6, Article 206, Nov. 2013, 11 pgs. [cited by applicant]
Goodfellow et al., “Deep Learning”, The MIT Press, Cambridge, Massachusetts, 2016, 26 pgs. [cited by applicant]
Graves, “Generating Sequences With Recurrent Neural Networks”, arXiv:1308.0850v1 [cs.NE], Aug. 4, 2013, 43 pgs. [cited by applicant]
Grzeszczuk et al., “NeuroAnimator: Fast Neural Network Emulation and Control of Physics-Based Models”, Computer Graphics Proceedings, Annual Conference Series, Orlando, Florida, Jul. 19-24, 1998, 12 pgs. [cited by applicant]
Ha et al., “A Neural Representation of Sketch Drawings”, arXiv:1704.03477v4 [cs.NE], May 19, 2017, 15 pgs. [cited by applicant]
He et al., “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification”, Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, Dec. 7-13, 2015, pp. 1026-1… [cited by applicant]
Hodgins et al., “Animating Human Athletics”, Proceedings of SIGGRAPH 95, Aug. 1995, pp. 71-78. [cited by applicant]
Holden et al., “Phase-Functioned Neural Networks for Character Control”, ACM Transactions on Graphics, vol. 36, No. 4, Article 42, Jul. 2017, 13 pgs. [cited by applicant]
Huang et al., “Full-Body Hybrid Motor Control for Reaching”, Proceedings of the 3rd International Conference on Motion in Games, Utrecht, The Netherlands, Nov. 14-16, 2010, pp. 36-47. [cited by applicant]
Ichim et al., “Phace: Physics-based Face Modeling and Animation”, ACM Transactions on Graphics, vol. 36, No. 4, Article 153, Jul. 2017, 14 pgs. [cited by applicant]
Jackson, “Address in Medicine: On the Comparative Study of Diseases of the Nervous System”, The British Medical Journal, vol. 2, Aug. 17, 1889, pp. 355-362. [cited by applicant]
Jongejan et al., “The Quick, Draw!—AI Experiment”, Google, Retrieved from: https://quickdraw.withgoogle.com/, 2016, 1 pg. [cited by applicant]
Kadlecek et al., “Reconstructing Personalized Anatomical Models for Physics-based Body Animation”, ACM Transactions on Graphics, vol. 35, No. 6, Article 213, Nov. 2016, 13 pgs. [cited by applicant]
Kahler et al., “Head Shop: Generating Animated Head Models with Anatomical Structure”, ACM SIGGRAPH Symposium on Computer Animation, Jul. 21, 2002, pp. 55-64. [cited by applicant]
Kingma et al., “Adam: A Method for Stochastic Optimization”, arXiv.org, Retrieved from: https://arxiv.org/abs/1412.6980v1 [cs.LG], Dec. 22, 2014, 9 pgs. [cited by applicant]
Latash, “The bliss (not the problem) of motor abundance (not redundancy)”, Experimental Brain Research, vol. 217, No. 1, Mar. 2012, 5 pgs. [cited by applicant]
Lee et al., “Comprehensive Biomechanical Modeling and Simulation of the Upper Body”, ACM Transactions on Graphics, Aug. 1, 2009, vol. 28, No. 4, pp. 1-17, XP055158375, ISSN: 0730-0301, DOI:10.1145/1559755. 1559756. [cited by applicant]
Lee et al., “Heads Up! Biomechanical Modeling and Neuromuscular Control of the Neck”, ACM Transactions on Graphics, vol. 25, No. 3, Jul. 2006, pp. 1188-1198. [cited by applicant]
Lee et al., “Locomotion Control for Many-Muscle Humanoids”, ACM Transactions on Graphics, vol. 33, No. 6, Nov. 2014, Article 218, 11 pgs. [cited by applicant]
Lee et al., “Realistic Modeling for Facial Animation”, In Computer Graphics Proceedings, Annual Conference Series, Los Angeles, California, Aug. 1995, pp. 55-62. [cited by applicant]
Lesmana et al., “A biologically inspired controller for fast eye movements”, Proceedings of the IEEE International Conference on Robotics and Automation, Shanghai, China, May 9-13, 2011. [cited by applicant]
Lesmana et al., “Active gaze stabilization”, Proceedings of the 2014 Indian Conference on Computer Vision Graphics and Image Processing, Bangalore, India, Dec. 14-18, 2014, No. 81, 8 pgs. [cited by applicant]
Liu et al., “Learning to Schedule Control Fragments for Physics-Based Characters Using Deep Q-Learning”, ACM Transactions on Graphics, vol. 36, No. 3, Article 29, Jun. 2017, 14 pgs. [cited by applicant]
Monheit et al., “A Kinematic Model of the Human Spine and Torso”, IEEE Computer Graphics and Applications, vol. 11, No. 2, Mar. 1991, pp. 29-38. [cited by applicant]
Nakada, “Deep Learning of Neuromuscular and Sensorimotor Control with Biomimetic Perception for Realistic Biomechanical Human Animation”, Dissertation, Jan. 1, 2017, XP055771117, University of California. [cited by applicant]
Nakada et al., “Deep Learning of Biomimetic Sensorimotor Control for Biomechanical Human Animation”, ACM Transactions on Graphics (TOG), Aug. 10, 2018, pp. 1-15, XP055801578, DOI: 10.1145/3197517.3201305. [cited by applicant]
Nakada et al., “Deep Learning of Biomimetic Visual Perception for Virtual Humans”, Proceedings of the 15th ACM Symposium on Applied Perception, Vancouver, British Columbia, Canada, Aug. 10-11, 2018, Article 20, 8 pgs. [cited by applicant]
Nakada et al., “Deep Learning of Neuromuscular and Visuomotor Control of a Biomimetic Simulated Humanoid”, IEEE Robotics and Automation Letters, IEEE, Feb. 10. 2020, vol. 5, No. 3, pp. 3952-3959, XP011787177, DOI: 10.11… [cited by applicant]
Nakada et al., “Deep Learning of Neuromuscular Control for Biomechanical Human Animation”, International Symposium on Visual Computing, Advances in Visual Computing, Part of the Lecture Notes in Computer Science Book Se… [cited by applicant]
Nakada et al., “Learning Biomimetic Perception for Human Sensorimotor Control”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018, pp. 1917-1922. [cited by applicant]
Nedel et al., “Anatomic modeling of deformable human bodies”, The Visual Computer, vol. 16, No. 6, Oct. 2000, pp. 306-321. [cited by applicant]
Peng et al., “DeepLoco: Dynamic Locomotion Skills Using Hierarchical Deep Reinforcement Learning”, ACM Transactions on Graphics, vol. 36, No. 4, Article 41, Jul. 2017, 16 pgs. [cited by applicant]
Rabie et al., “Active Perception in Virtual Humans”, In Proceedings of the Vision Interface, Montreal, Canada, 2000, 7 pgs. [cited by applicant]
Sachdeva et al., “Biomechanical Simulation and Control of Hands and Tendinous Systems”, ACM Transactions on Graphics, vol. 34, No. 4, Aug. 2015, Article 42, 10 pgs. [cited by applicant]
Scheepers et al., “Anatomy-Based Modeling of the Human Musculature”, Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, 1997, pp. 163-172. [cited by applicant]
Scholz et al., “The uncontrolled manifold concept: identifying control variables for a functional task”, Experimental Brain Research, vol. 126, No. 3, May 1999, pp. 289-306. [cited by applicant]
Schwartz, “Spatial Mapping in the Primate Sensory Projection: Analytic Structure and Relevance to Perception”, Biological Cybernetics, vol. 25, No. 4, Dec. 1977, pp. 181-194. [cited by applicant]
Shao et al., “A general joint component framework for realistic articulation in human characters”, Proceedings of the 2003 Symposium on Interactive 3D Graphics, Monterey, California, Apr. 27-30, 2003, pp. 11-18. [cited by applicant]
Si et al., “Realistic Biomechanical Simulation and Control of Human Swimming”, ACM Transactions on Graphics, vol. 34, No. 1, Nov. 2014, Article 10, 15 pgs. [cited by applicant]
Sifakis et al., “Automatic Determination of Facial Muscle Activations from Sparse Motion Capture Marker Data”, ACM Transactions on Graphics, vol. 24, No. 3, Jul. 2005, pp. 417-425. [cited by applicant]
Simhon et al., “Sketch Interpretation and Refinement Using Statistical Models”, Proceedings of the 15th Eurographics Conference on Rendering Techniques, Norrköping, Sweden, Jun. 21-23, 2004, 10 pgs. [cited by applicant]
Sueda et al., “Musculotendon Simulation for Hand Animation”, ACM Transactions on Graphics, vol. 27, No. 3, Article 83, Aug. 2008, 8 pgs. [cited by applicant]
Sun et al., “Robot learns Chinese calligraphy from Demonstrations”, Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Chicago, Illinois, Sep. 14-18, 2014. [cited by applicant]
Terzopoulos et al., “Animat vision: Active vision in artificial animals”, Proceedings of IEEE International Conference on Computer Vision, Cambridge, Massachusetts, Jun. 20-23, 1995. [cited by applicant]
Terzopoulos et al., “Physically-Based Facial Modeling, Analysis, and Animation”, Journal of Visualization and Computer Animation, vol. 1, No. 2, Dec. 1990, pp. 73-80. [cited by applicant]
Tresset et al., “Portrait drawing by Paul the robot”, Computers & Graphics, vol. 37, No. 5, Aug. 2013, pp. 348-363. [cited by applicant]
Wang et al., “Optimizing Locomotion Controllers Using Biologically-Based Actuators and Objectives”, ACM Transactions on Graphics, vol. 31, No. 4, Article 25, Jul. 2012, 11 pgs. [cited by applicant]
Wei et al., “Biomechanical simulation of human eye movement”, Proceedings of the 5th International Conference on Biomedical Simulation, Phoenix, Arizona, Jan. 23-24, 2010, pp. 108-118. [cited by applicant]
Wilhelms et al., “Anatomically Based Modeling”, Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, 1997, pp. 173-180. [cited by applicant]
Yeo et al., “Eyecatch: Simulating Visuomotor Coordination for Object Interception”, ACM Transactions on Graphics, vol. 31, No. 4, 2012, pp. 1-10. [cited by applicant]
Zhang et al., “Drawing and Recognizing Chinese Characters with Recurrent Neural Network”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, No. 4, Apr. 18, 2017, pp. 849-862. [cited by applicant]
Zordan et al., “Breathe Easy: Model and control of simulated respiration for animation”, Proceedings of the 2004 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Grenoble, France, Aug. 27-29, 2004, 9 pgs. [cited by applicant]