Markerless motion capture system
View Patent ↗A markerless motion capture system is provided for measurements accurate enough for biomechanical, clinical, sport, entertainment, animation, game and movie, design, ergonomics, surveillance applications. The system has multiple cameras distributed around a viewing volume. The cameras allow for the creation of three-dimensional mesh representations of an object dynamically moving within the viewing volume. A model of the object that incorporates specific morphological and kinematic model information (including soft joint constraints) is then matched to the captured three-dimensional mesh representations. The matching routine aims to embed the model into each of the three-dimensional representations using (i) iterative closest point or simulated annealing algorithms and (ii) using soft joint constraints. This unique combination of routines offers a simple, time-efficient, accurate and thus more meaningful assessment of movements. The system further offers feasibility of accurately and precisely measuring three-dimensional kinematics of the dynamically moving object or human.
1. A method of generating a three-dimensional dynamic model of a single subject moving within a volume, comprising:
(a) defining a static full-body model R of said single subject, wherein said model R is a three-dimensional articulated multi-segment model defined by polygon meshes for the body segments, wherein said articulation is defined by three-dimensional joint centers with soft joint constraints between the neighboring body segments in said articulated multi-segment model providing up to 6 degrees of freedom (DOF) in articulation at each joint;
(b) markerless motion capturing, with a multiple camera system, a three-dimensional representation of said single subject moving within said volume;
(c) defining for each of said markerless-motion-captured frames a three-dimensional full-body model C of said single subject;
(d) matching said full-body model C of each of said markerless-motion-captured frames with said full-body model R, wherein said matching allows for deviations between the neighboring body segment using soft joint constraints; and
(e) outputting for each of said markerless-motion-captured frames a full-body three-dimensional model of said single subject for the movement within said volume, wherein said full-body three-dimensional model is a three-dimensional articulated multi-segment model with three-dimensional soft-joint constraints providing up to 6 DOF in articulation at each joint.
2. The method as set forth in claim 1 , wherein said model C is a three-dimensional polygon mesh, a three-dimensional points cloud, a set of silhouettes or features characterizing the human body moving within said viewing volume.
3. The method as set forth in claim 1 , wherein the deviation in said soft joint constraints varies from at least 0.1 mm or more.
4. The method as set forth in claim 1 , wherein said matching further comprises initializing the first frame of said matching using a stochastic optimization method.
5. The method as set forth in claim 1 , wherein said matching further comprises initializing the first frame of said matching using a gradient descent method.
6. The method as set forth in claim 1 , wherein said matching further comprises initializing the first frame of said matching using alignment of said model C and said model R based on the body segment orientations and centers of mass.
7. The method as set forth in claim 1 , wherein said matching further comprises initializing the first frame of said matching using the model R and model C inertia tensors.
8. The method as set forth in claim 1 , wherein said matching is based on evolutions of an iterative process of iterative closet point (ICP) or simulated annealing (SA) methods.
9. The method as set forth in claim 1 , wherein said three-dimensional full-body model R and model C are defined by a homogenous distribution of points of the respective meshes.
10. The method as set forth in claim 1 , wherein each body segment of said model R has it own three-dimensional local reference frame.
11. The method as set forth in claim 1 , wherein each body segment of said model R has a global three-dimensional reference frame with said volume or said environment.
12. The method as set forth in claim 1 , wherein each body segment of said model R has it own three-dimensional local reference frame, and wherein each of said local reference frames defined on said matched full-body model R are used to provide up to 6 DOF in articulation at each joint.
13. A computer-readable non-transitory storage medium containing a set of instructions executable on a computer, said set of instructions comprising:
(a) a reference model definition routine for defining a static full-body model R of said single subject, wherein said model R is a three-dimensional articulated multi-segment model defined by polygon meshes for the body segments, wherein said articulation is defined by three-dimensional joint centers with soft joint constraints between the neighboring body segments in said articulated multi-segment model providing up to 6 degrees of freedom (DOF) in articulation at each joint;
(b) a markerless motion capture routine for markerless motion capturing, with a multiple camera system, a three-dimensional representation of said single subject moving within said volume;
(c) a definition routine for defining for each of said markerless-motion-captured frames a three-dimensional full-body model C of said single subject;
(d) a matching routine for matching said full-body model C of each of said markerless-motion-captured frames with said full-body model R, wherein said matching allows for deviations between the neighboring body segment using soft joint constraints; and
(e) an output routine for outputting for each of said markerless-motion-captured frames a full-body three-dimensional model of said single subject for the movement within said volume, wherein said full-body three-dimensional model is a three-dimensional articulated multi-segment model with three-dimensional soft-joint constraints providing up to 6 DOF in articulation at each joint.
14. The computer-readable non-transitory storage medium as set forth in claim 13 , wherein said model C is a three-dimensional polygon mesh, a three-dimensional points cloud, a set of silhouettes or features characterizing the human body moving within said viewing volume.
15. The computer-readable non-transitory storage medium as set forth in claim 13 , wherein the deviation in said soft joint constraints varies from at least 0.1 mm or more.
16. The computer-readable non-transitory storage medium as set forth in claim 13 , wherein said matching further comprises initializing the first frame of said matching using a stochastic optimization method.
17. The computer-readable non-transitory storage medium as set forth in claim 13 , wherein said matching further comprises initializing the first frame of said matching using a gradient descent method.
18. The computer-readable non-transitory storage medium as set forth in claim 13 , wherein said matching further comprises initializing the first frame of said matching using alignment of said model C and said model R based on the body segment orientations and centers of mass.
19. The computer-readable non-transitory storage medium as set forth in claim 13 , wherein said matching further comprises initializing the first frame of said matching using the model R and model C inertia tensors.
20. The computer-readable non-transitory storage medium as set forth in claim 13 , wherein said matching is based on evolutions of an iterative process of iterative closet point (ICP) or simulated annealing (SA) methods.
21. The computer-readable non-transitory storage medium as set forth in claim 13 , wherein said three-dimensional full-body model R and model C are defined by a homogenous distribution of points of the respective meshes.
22. The computer-readable non-transitory storage medium as set forth in claim 13 , wherein each body segment of said model R has it own three-dimensional local reference frame.
23. The computer-readable non-transitory storage medium as set forth in claim 13 , wherein each body segment of said model R has a global three-dimensional reference frame with said volume or said environment.
24. The computer-readable non-transitory storage medium as set forth in claim 13 , wherein each body segment of said model R has it own three-dimensional local reference frame, and wherein each of said local reference frames defined on said matched full-body model R are used to provide up to 6 DOF in articulation at each joint.