Reconstruction of articulated objects from a moving camera

1. A method of reconstructing an articulated object from images captured with a moving camera, the method comprising:

generating a plurality of 2D tracks, wherein at least one of the 2D tracks of the plurality of 2D tracks comprises a plurality of 2D feature points, wherein one of the 2D feature points identifies a two-dimensional location of a unique one of a series of points on an articulated object in a one of a sequenced series of captured images, wherein the sequenced series of captured images are captured by a moving camera, and wherein the moving camera is moving with respect to at least a fixed portion of a background of the images forming the sequenced series of captured images;

generating a manifold of rays comprising a plurality of rays, wherein the rays of the manifold of rays are defined in part as extending through a unique one of the data points of one of the 2D tracks;

converting at least some of the plurality of 2D data points of the plurality of 2D tracks to 3D data points by identifying a position for at least one of the plurality of 2D data points along one of the plurality of rays forming the manifold of rays; and

generating a first 3D representation of the articulated object for each of the images of the sequenced series of captured images, wherein the first 3D representation comprises the 3D data points converted from 2D data points gathered from one of the images of the sequenced series of captured images.

2. The method of claim 1 , wherein the plurality of 2D data points together identify a 2D location of the unique one of the series of points of the articulated object in each of the sequenced series of captured images.

3. The method of claim 1 , wherein generating the manifold of rays comprises:

identifying a lens center for a lens of the moving camera for each of the plurality of the images forming the sequenced series of captured images; and

generating a plurality of rays, wherein a ray extends through a pair of a lens center for one of the images forming the sequenced series of captured images and at least one 2D feature point for that one of the images forming the sequenced series of captured images.

4. The method of claim 3 , wherein a unique ray is generated for each pair of one lens center and one 2D feature point for each of the images forming the sequenced series of images.

5. The method of claim 3 , wherein generating a manifold of rays comprises generating a plurality of manifolds or rays, wherein one of manifolds of rays forming the plurality of manifolds of rays is generated for each of the plurality of 2D tracks.

6. The method of claim 5 , wherein identifying a position for at least some of the plurality of 2D data points along the each of the plurality of rays forming the manifold of rays comprises:

generating a first energy function the result of which is based on the magnitude of difference in the positions of temporally adjacent 3D data points along their respective ray; and

adjusting the position of the 3D data points to minimize the result of the first energy function.

7. The method of claim 6 , further comprising refining the first 3D representation of the articulated object by applying piecewise rigidity constraints, wherein applying piecewise rigidity constrains comprises:

identifying first and second groups of 3D data points as belonging to common first and second rigid members;

generating first and second rigid objects corresponding to the identified groups of 3D data points belonging to the first and second common rigid members;

determining the rotation and translation of the first and second rigid objects from one image of the sequenced series of captured images to a temporally adjacent image of the sequence of captured images;

generating a second 3D representation of the articulated object for each of the images based on the generated first and second rigid objects and the determined rotation and translation of the generated first and second rigid objects;

determining the error between the second 3D representation and the 2D tracks; and

refining the determined rotation and translation of the first and second rigid objects to minimize the error between the second 3D representation and the 2D tracks.

8. The method of claim 7 , wherein refining the determined rotation and translation of the first and second rigid objects to minimize the error between the second 3D representation and the 2D tracks comprises adjusting the determined error with a smoothness parameter.

9. The method of claim 7 , further comprising generating a third 3D representation of the articulated object by applying kinematic constraints to the second 3D representation.

10. The method of claim 9 , wherein applying kinematic constraints comprises:

determining the distance between the first and second rigid members for each image of the sequenced series of captured images;

determining the variation in the determined distances between the first and second rigid members; and

identifying a joint if the determined distance between the first and second rigid members and the variation in the determined distances between the first and second rigid members indicate the presence of a joint.

11. The method of claim 9 , wherein applying kinematic constraints comprises:

determining the position of the first rigid member based on the rotation and translation of the first rigid member;

determining the position of the second rigid member based on the rotation and translation of the second rigid member and the determined position of the first rigid member;

generating the third 3D representation based on the determined position of the first and second rigid members;

determining the error between the third 3D representation and the 3D tracks;

refining the determined positions of the first and second rigid members to minimize the error between the third 3D representation and the 2D tracks.

12. A system for reconstructing an articulated object from images captured with a moving camera, the method comprising:

memory comprising stored instructions; and

a processor configured to:

generate a plurality of 2D tracks, wherein at least one of the 2D tracks of the plurality of 2D tracks comprises a plurality of 2D data points identifying 2D locations of a unique one of a series of points on an articulated object in a one of a sequenced series of captured images, wherein the sequenced series of captured images are captured by a moving camera, and wherein the moving camera is moving with respect to at least a fixed portion of a background of the images forming the sequenced series of captured images;

generate a manifold of rays comprising a plurality of rays, wherein the rays of the manifold of rays are defined in part as extending through a unique one of the data points of one of the 2D tracks;

convert at least some of the plurality of 2D data points of the plurality of 2D tracks to 3D data points by identifying a position for at least some of the plurality of 2D data points along the each of the plurality of rays forming the manifold of rays; and

generate a first 3D representation of the articulated object for each of the images of the sequenced series of captured images, wherein the first 3D representation comprises the 3D data points converted from 2D data points gathered from one of the images of the sequenced series of captured images.

13. The system of claim 12 , wherein the plurality of 2D data points together identify a 2D location of the unique one of the series of points of the articulated object in each of the sequenced series of captured images.

14. The system of claim 12 , wherein generating the manifold of rays comprises:

identifying a lens center for a lens of the moving camera for each of the plurality of the images forming the sequenced series of captured images; and

generating a plurality of rays, wherein a ray extends through a pair of a lens center for one of the images forming the sequenced series of captured images and the 2D data point for that one of the images forming the sequenced series of captured images.

15. The system of claim 14 , wherein a unique ray is generated for each pair of one lens center and one 2D data point for each of the images forming the sequenced series of images.

16. The system of claim 15 , wherein generating a manifold of rays comprises generating a plurality of manifolds or rays, wherein one of manifolds of rays forming the plurality of manifolds of rays is generated for each of the plurality of 2D tracks.

17. The system of claim 16 , wherein identifying a position for at least some of the plurality of 2D data points along the each of the plurality of rays forming the manifold of rays comprises:

generating a first energy function the result of which is based on the magnitude of difference in the positions of temporally adjacent 3D data points along their respective ray; and

adjusting the position of the 3D data points to minimize the result of the first energy function.

18. The system of claim 17 , wherein the processor is further configured to refine the first 3D representation of the articulated object by applying piecewise rigidity constraints, wherein applying piecewise rigidity constrains comprises:

identifying first and second groups of 3D data points as belonging to common first and second rigid members;

generating first and second rigid objects corresponding to the identified groups of 3D data points belonging to the first and second common rigid members;

determining the rotation and translation of the first and second rigid objects from one image of the sequenced series of captured images to a temporally adjacent image of the sequence of captured images;

generating a second 3D representation of the articulated object for each of the images based on the generated first and second rigid objects and the determined rotation and translation of the generated first and second rigid objects;

determining the error between the second 3D representation and the 2D tracks; and

refining the determined rotation and translation of the first and second rigid objects to minimize the error between the second 3D representation and the 2D tracks.

19. The system of claim 18 , wherein refining the determined rotation and translation of the first and second rigid objects to minimize the error between the second 3D representation and the 2D tracks comprises adjusting the determined error with a smoothness parameter.

20. The system of claim 19 , wherein the processor is further configured to generate a third 3D representation of the articulated object by applying kinematic constraints to the second 3D representation.

21. The system of claim 20 , wherein applying kinematic constraints comprises:

determining the distance between the first and second rigid members for each image of the sequenced series of captured images;

determining the variation in the determined distances between the first and second rigid members; and

identifying a joint if the determined distance between the first and second rigid members and the variation in the determined distances between the first and second rigid members indicate the presence of a joint.

22. The system of claim 20 , wherein applying kinematic constraints comprises:

determining the position of the first rigid member based on the rotation and translation of the first rigid member;

determining the position of the second rigid member based on the rotation and translation of the second rigid member and the determined position of the first rigid member;

generating the third 3D representation based on the determined position of the first and second rigid members;

determining the error between the third 3D representation and the 3D tracks; and

refining the determined positions of the first and second rigid members to minimize the error between the third 3D representation and the 2D tracks.