Motion-controlled body capture and reconstruction
A method of generating unrecorded camera views may include receiving a plurality of 2-D video sequences of a subject in a real 3-D space, where each 2-D video sequence may depict the subject from a different perspective. The method may also include generating a 3-D representation of the subject in a virtual 3-D space, where a geometry and texture of the 3-D representation may be generated based on the 2D video sequences, and the motion of the 3-D representation in the virtual 3-D space is based on motion of the subject in the real 3-D space. The method may additionally include generating a 2-D video sequence of the motion of the 3D representation using a virtual camera in the virtual 3-D space where the perspective of the virtual camera may be different than the perspectives of the plurality of 2-D video sequences.
1. A method comprising:
receiving a plurality of 2-D video sequences of a subject in a real 3-D space, wherein each 2-D video sequence in the plurality of 2-D video sequences depicts the subject from a different perspective;
generating a 3-D representation of the subject in a virtual 3-D space, wherein:
a geometry and texture of the 3-D representation is generated based on the plurality of 2-D video sequences;
a motion of the 3-D representation in the virtual 3-D space is based on a motion of the subject in the real 3-D space; and
an outer surface of the 3-D representation is missing information, and the texture of the 3-D representation is missing texture information;
generating a full 3-D representation of the subject in the virtual 3-D space by at least combining the 3-D representation of the subject with an existing 3-D digital model, wherein:
the 3-D digital model is generated using a 3-D scan of a volume of the subject in the real 3-D space, and the 3-D scan is of the same subject that was captured in the 2-D video sequence; and
the 3-D digital model is used to fill in the missing information of the 3-D representation by:
smoothing the outer surface of the 3-D representation;
filling volumetric holes in the 3-D representation that were not visible in the plurality of 2-D video sequences; and
properly aligning or dimensioning the 3-D representation;
generating the missing texture information using corresponding pixels in prior or subsequent frames of a motion of the of the full 3-D representation; and
generating a 2-D video sequence of the motion of the full 3-D representation using a virtual camera in the virtual 3-D space, wherein a perspective of the virtual camera is different than the perspectives of the plurality of 2-D video sequences.
2. The method of claim 1 , wherein the geometry of the 3-D representation is generated using a machine vision system comprising an infrared projector and a machine vision camera.
3. The method of claim 1 , wherein generating the 3-D representation of the subject in the virtual 3-D space comprises generating a 2-D cutout silhouette for each frame in the 2-D video sequences, and combining the 2-D cutout silhouettes to generate a volumetric 3-D shell of the subject.
4. The method of claim 1 , wherein the texture of the 3-D representation is based on a high definition camera system wherein a plurality of cameras track the motion of the subject through the real 3-D space.
5. The method of claim 1 , further comprising equalizing the lighting on the 3-D representation of the subject, and re-lighting the 3-D representation of the subject in the virtual 3-D space.
6. The method of claim 1 , wherein the texture of the 3-D representation is generated using view-dependent texture reconstruction that uses pixels from real-world cameras that are most relevant to the view of the virtual camera.
7. The method of claim 1 , further comprising receiving position information from a plurality of cameras used to capture the plurality of 2-D video sequences, and sending commands configured to calibrate the plurality of cameras.
8. A system comprising:
one or more processors; and
a non-transitory storage memory comprising instructions which, when executed by the one or more processors, cause the one or more processors to perform operations including:
receiving a plurality of 2-D video sequences of a subject in a real 3-D space, wherein each 2-D video sequence in the plurality of 2-D video sequences depicts the subject from a different perspective;
generating a 3-D representation of the subject in a virtual 3-D space, wherein:
a geometry and texture of the 3-D representation is generated based on the plurality of 2-D video sequences;
a motion of the 3-D representation in the virtual 3-D space is based on a motion of the subject in the real 3-D space; and
an outer surface of the 3-D representation is missing information, and the texture of the 3-D representation is missing texture information;
generating a full 3-D representation of the subject in the virtual 3-D space by at least combining the 3-D representation of the subject with an existing 3-D digital model, wherein:
the 3-D digital model is generated using a 3-D scan of a volume of the subject in the real 3-D space, and the 3-D scan is of the same subject that was captured in the 2-D video sequence; and
the 3-D digital model is used to fill in the missing information of the 3-D representation by:
smoothing the outer surface of the 3-D representation;
filling volumetric holes in the 3-D representation that were not visible in the plurality of 2-D video sequences; and
properly aligning or dimensioning the 3-D representation;
generating the missing texture information using corresponding pixels in prior or subsequent frames of a motion of the of the full 3-D representation; and
generating a 2-D video sequence of the motion of the full 3-D representation using a virtual camera in the virtual 3-D space, wherein a perspective of the virtual camera is different than the perspectives of the plurality of 2-D video sequences.
9. The system of claim 8 , wherein the geometry of the 3-D representation is generated using a machine vision system comprising an infrared projector and a machine vision camera.
10. The system of claim 8 , wherein generating the 3-D representation of the subject in the virtual 3-D space comprises generating a 2-D cutout silhouette for each frame in the 2-D video sequences, and combining the 2-D cutout silhouettes to generate a volumetric 3-D shell of the subject.
11. The system of claim 8 , wherein the texture of the 3-D representation is based on a high-definition camera system wherein a plurality of cameras track the motion of the subject through the real 3-D space.
12. The system of claim 8 , wherein the instructions cause the one or more processors to perform operations further comprising:
equalizing the lighting on the 3-D representation of the subject, and re-lighting the 3-D representation of the subject in the virtual 3-D space.
13. The system of claim 8 , wherein the texture of the 3-D representation is generated using view-dependent texture reconstruction that uses pixels from real-world cameras that are most relevant to the view of the virtual camera.
14. The system of claim 8 , wherein the instructions cause the one or more processors to perform operations further comprising:
receiving position information from a plurality of cameras used to capture the plurality of 2-D video sequences, and sending commands configured to calibrate the plurality of cameras.
15. A non-transitory storage medium comprising instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving a plurality of 2-D video sequences of a subject in a real 3-D space, wherein each 2-D video sequence in the plurality of 2-D video sequences depicts the subject from a different perspective;
generating a 3-D representation of the subject in a virtual 3-D space, wherein:
a geometry and texture of the 3-D representation is generated based on the plurality of 2-D video sequences;
a motion of the 3-D representation in the virtual 3-D space is based on a motion of the subject in the real 3-D space; and
an outer surface of the 3-D representation is missing information, and the texture of the 3-D representation is missing texture information;
generating a full 3-D representation of the subject in the virtual 3-D space by at least combining the 3-D representation of the subject with an existing 3-D digital model, wherein:
the 3-D digital model is generated using a 3-D scan of a volume of the subject in the real 3-D space, and the 3-D scan is of the same subject that was captured in the 2-D video sequence; and
the 3-D digital model is used to fill in the missing information of the 3-D representation by:
smoothing the outer surface of the 3-D representation;
filling volumetric holes in the 3-D representation that were not visible in the plurality of 2-D video sequences; and
properly aligning or dimensioning the 3-D representation;
generating the missing texture information using corresponding pixels in prior or subsequent frames of a motion of the of the full 3-D representation; and
generating a 2-D video sequence of the motion of the full 3-D representation using a virtual camera in the virtual 3-D space, wherein a perspective of the virtual camera is different than the perspectives of the plurality of 2-D video sequences.
16. The storage medium of claim 15 , wherein the geometry of the 3-D representation is generated using a machine vision system comprising an infrared projector and a machine vision camera.
17. The storage medium of claim 15 , wherein generating the 3-D representation of the subject in the virtual 3-D space comprises generating a 2-D cutout silhouette for each frame in the 2-D video sequences, and combining the 2-D cutout silhouettes to generate a volumetric 3-D shell of the subject.