Systems, methods and devices for motion capture using video imaging

1. A processor-implemented method comprising:

generating a digital representation (R) of a figure by correlating a visual image of the figure with three-dimensional information about the figure, the digital representation of the figure including a two-dimensional surface mesh of the figure and a volumetric representation of the figure using a plurality of three-dimensional shapes;

finding, using markerless capture, reference points that are common to each of two temporally distinct visual images of the figure, the reference points representing movement of the figure between the two images;

correlating the reference points with vertices in the volumetric representation;

deforming, in response to positional constraints placed on the movement of the plurality of three-dimensional shapes between the two images, the volumetric representation of the figure as a function of the reference points and the correlated vertices to allow for characterization of three-dimensional movement of the plurality of three-dimensional shapes;

adjusting the deformation of the volumetric representation of the figure in response to a comparison of silhouette data from one of the two images to the volumetric representation;

deforming the surface mesh of the figure as a function of the volumetric deformation by mapping the deformation of the volumetric representation to the surface mesh based upon correlation between individual points in the surface mesh to multiple three-dimensional shapes of the plurality of three-dimensional shapes; and

generating, responsive to the deformed two-dimensional surface mesh and to the deformed volumetric representation of the figure, an updated digital representation of the figure.

2. The method of claim 1 , wherein the surface mesh includes a set of triangle shapes and the plurality of three-dimensional shapes includes tetrahedral shapes.

3. The method of claim 2 , wherein the step of deforming the volumetric representation includes, for each tetrahedral shape, freedom of movement in three dimensions that is independent of a skeletal model of the figure.

4. The method of claim 1 , wherein the step of generating a digital representation (R) further includes the step of generating the digital representation by:

characterizing a dense set of three-dimensional points (P) on a surface of R and connectivity between the points on the surface;

defining the figure via a set of coordinates for each point in P; and

characterizing a sparse volumetric version (V) of R representative of the shape of the figure in a reference pose, wherein V is represented as a set of tetrahedron vertices (Q) and poses are defined by a three-dimensional coordinate for each point in Q.

5. The method of claim 4 , wherein the step of deforming the surface mesh of the figure further includes using a second set of point constraints constructed from a set of reference points different from the reference points from in the step of deforming the volumetric representation of the figure.

6. The method of claim 4 , wherein the step of generating an updated digital representation of the figure includes the steps of:

defining, responsive to the deformed surface mesh and to the deformed volumetric representation of the figure, positional information about three-dimensional point constraints;

generating, responsive to the defined positional information, a captured performance model (M) representative of the geometry of the figure at respective times, wherein M includes a set of three-dimensional point positions for all points P of R at each respective time and a set of positions for each vertex Q of V; and

generating a representation (I) of an appearance at each time step corresponding to measurements from image sensors.

7. The method of claim 6 , wherein the step of generating a representation (I) includes the step of using data from the image sensors including:

time-varying multi-view image data of the figure, features describing correspondence between surface points of the figure for temporally-different image data, and features describing a posture and a detailed surface appearance of the figure at each time step.

8. The method of claim 6 , further including the steps of

capturing a coarse pose of the figure at each time step by finding an optimal deformation for V in view of I; and

generating the model M by tracking the figure using a coarse pose estimate to compute a detailed surface estimate at each time step of a performance using a fine surface model of R.

9. The method of claim 4 , further including the steps of

estimating an animation model M using measurements from image sensors and without use of fiducial markers;

and estimating parameters of M by alignment with a representation of a real world subject I.

10. The method of claim 1 , further including passively reconstructing spatio-temporally coherent shape, motion and texture for a sequence of temporally distinct images.

11. The method of claim 1 , wherein the step of deforming the volumetric representation includes preserving the shape of the digital representation locally such that the volume is preserved locally.

12. The method of claim 1 , further including the steps of

tracking at low-resolution to find reference points; and

tracking at high-resolution to apply a fine deformation.

13. The method of claim 1 , further including the steps of

representing the figure as a set of three-dimensional shapes having a set of vertices; and

selecting a subset of the set of vertices as a function of a projected position of the set of vertices and a distance between the projected position and an actual position of the set of vertices.

14. The method of claim 1 , wherein the step of applying a volumetric deformation includes accounting for non-rigidly deforming portions of the figure, the non-rigidly deforming portions moving substantially independently from rigidly deforming portions of the figure.

15. The method of claim 1 , wherein the step of applying a volumetric deformation includes accounting for non-rigidly deforming portions of the figure, the non-rigidly deforming portions moving substantially independently from rigidly deforming portions of the figure by removing influences of a linear Laplacian deformation.

16. A device comprising:

a processing circuit configured and arranged for

generating a digital representation (R) of a figure by correlating a visual image of the figure with three-dimensional information about the figure, the digital representation of the figure including a surface mesh of the figure and a volumetric representation of the figure using a plurality of three-dimensional shapes;

finding, using markerless capture, reference points that are common to each of two temporally distinct visual images of the figure, the reference points representing movement of the figure between the two images;

correlating the reference points with vertices in the volumetric representation;

deforming, in response to positional constraints placed on the movement of the plurality of three-dimensional shapes between the two images, the volumetric representation of the figure as a function of the reference points and the correlated vertices to allow for characterization of three-dimensional movement of the plurality of three-dimensional shapes;

adjusting the deformation of the volumetric representation of the figure in response to a comparison of silhouette data from one of the two images to the volumetric representation;

deforming the surface mesh of the figure as a function of the volumetric deformation by mapping the deformation of the volumetric representation to the surface mesh based upon correlation between individual points in the surface mesh to multiple three-dimensional shapes of the plurality of three-dimensional shapes; and

generating, responsive to the deformed surface mesh and to the deformed volumetric representation of the figure, an updated digital representation of the figure.

17. The device of claim 16 , wherein the processing circuit is further for

using a low-resolution tracking approach to find reference points; and

using a high-resolution approach to apply the fine deformation.

18. The device of claim 17 , wherein the processing circuit is further for determining a pose for the figure using a lower-detail model derived from the low-resolution tracking approach.

19. The device of claim 16 , wherein the processing circuit is further for passively reconstructing spatio-temporally coherent shape, motion and texture for a sequence of temporally distinct images.

20. The device of claim 16 , wherein the processing circuit is further for volumetrically deforming the volumetric representation of the figure independent from a skeleton of the figure and with, for each three-dimensional shape, freedom of movement in three dimensions.

21. The device of claim 16 , wherein the processing circuit is further for volumetrically deforming the digital representation using a tetrahedral mesh modeling the figure.

22. The device of claim 16 , wherein the processing circuit is further for

volumetrically deforming the volumetric representation of the figure in compliance with three-dimensional point constraints, and for

deforming the surface mesh in compliance with a volumetric model pose and additional three-dimensional point constraints.

23. The device of claim 16 , wherein the processing circuit is further for selecting vertices from a set of all vertices of the three-dimensional shape, as a function of a projected position of the set of all vertices and a distance between the projected position and an actual position of the set of all vertices.

24. The device of claim 16 , wherein the processing circuit is further for, during the step of deforming the volumetric representation, accounting for non-rigid deforming portions of the figure, the non-rigid deforming portions of the figure moving substantially independent from movement of rigidly deforming portions of the figure.

25. A non-transitory-storage medium containing stored data that when accessed causes a processing circuit to perform the steps of:

generating a digital representation (R) of a figure by correlating a visual image of the figure with three-dimensional information about the figure, the digital representation of the figure including a surface mesh of the figure and a volumetric representation of the figure using a plurality of three-dimensional shapes;

finding, using markerless capture, reference points that are common to each of two temporally distinct visual images of the figure, the reference points representing movement of the figure between the two images;

correlating the reference points with vertices in the volumetric representation;

deforming, in response to positional constraints placed on the movement of the plurality of three-dimensional shapes between the two images, the volumetric representation of the figure as a function of the reference points and the correlated vertices to allow for characterization of three-dimensional movement of the plurality of three-dimensional shapes;

adjusting the deformation of the volumetric representation of the figure in response to a comparison of silhouette data from one of the two images to the volumetric representation;

deforming the surface mesh of the figure as a function of the volumetric deformation by mapping the deformation of the volumetric representation to the surface mesh based upon correlation between individual points in the surface mesh to multiple three-dimensional shapes of the plurality of three-dimensional shapes; and

generating, responsive to the deformed surface mesh and to the deformed volumetric representation of the figure, an updated digital representation of the figure.

26. The storage medium of claim 25 , wherein the stored data when accessed further causes a processing circuit to perform the steps of

volumetrically deforming the volumetric representation of the figure in compliance with three-dimensional point constraints, and

deforming the surface mesh in compliance with a volumetric model pose and additional three-dimensional point constraints.

27. The storage medium of claim 25 , wherein the stored data when accessed further causes a processing circuit to perform the step of preserving a local shape for maintaining the volume of the figure.