Parts-based decomposition of human body for blend shape pipeline integration and muscle prior creation
View Patent ↗Parts-based blendshape generation involves establishing a 4D capture sequence. A face, hands and legs are able to be established for the meshes using a template to generate tracked and templated meshes. Specific muscle deformations are extracted. Pose-specific spatial deformations are integrated into a pose to generate specific flesh deformations. A muscle deformation method enables muscle part-based approach deformation (e.g., just biceps if flexing arms). The muscle deformation also enables generating a novel pose not captured.
1 . A method programmed in a non-transitory of a device comprising:
acquiring 4D scanned Range of Motion (ROM) shapes;
tracking the ROM shapes; and
performing a deformation transfer specific to a pose based on vertex correspondences, wherein the vertex correspondences are based on muscle groupings, wherein one or more inconsistencies between meshes in tracked keyframes are removed after acquiring the ROM shapes by using templated body parts and removing one or more deformations in non-desired parts to generate a semantic mesh with deformation mapped to skeleton joints directly for learning body part deformations, wherein mapping is based on a previous capture of range of motion of muscles.
2 . The method of claim 1 wherein a base shape is segmented, and the ROM shapes are tracked to the segmented base shape.
3 . The method of claim 1 wherein a desired base shape is used for tracking, and the resulting tracked ROM shapes are semantically segmented.
4 . An apparatus comprising:
a non-transitory memory for storing an application, the application for:
acquiring 4D scanned Range of Motion (ROM) shapes;
tracking the ROM shapes; and
performing a deformation transfer specific to a pose based on vertex correspondences, wherein the vertex correspondences are based on muscle groupings, wherein one or more inconsistencies between meshes in tracked keyframes are removed after acquiring the ROM shapes by using templated body parts and removing one or more deformations in non-desired parts to generate a semantic mesh with deformation mapped to skeleton joints directly for learning body part deformations, wherein mapping is based on a previous capture of range of motion of muscles; and
a processor coupled to the memory, the processor configured for processing the application.
5 . The apparatus of claim 4 wherein a base shape is segmented, and the ROM shapes are tracked to the segmented base shape.
6 . The apparatus of claim 4 wherein a desired base shape is used for tracking, and the resulting tracked ROM shapes are semantically segmented.
7 . A system comprising:
a volumetric capture system for 3D and 4D scanning including capturing photos and video simultaneously, wherein the 3D scanning and 4D scanning includes detecting muscle deformation of an actor; and
a computing device configured for:
acquiring 4D scanned Range of Motion (ROM) shapes;
tracking the ROM shapes; and
performing a deformation transfer specific to a pose based on vertex correspondences, wherein the vertex correspondences are based on muscle groupings, wherein one or more inconsistencies between meshes in tracked keyframes are removed after acquiring the ROM shapes by using templated body parts and removing one or more deformations in non-desired parts to generate a semantic mesh with deformation mapped to skeleton joints directly for learning body part deformations, wherein mapping is based on a previous capture of range of motion of muscles.
8 . The system of claim 7 wherein a base shape is segmented, and the ROM shapes are tracked to the segmented base shape.
9 . The system of claim 7 wherein a desired base shape is used for tracking, and the resulting tracked ROM shapes are semantically segmented.
10 . A method programmed in a non-transitory of a device comprising:
acquiring a rest pose including a mesh sequence including pose modeling and registration for blending, wherein the mesh sequence is acquired using an integrated photo-video volumetric capture system for 3D/4D scans by acquiring images and videos of a subject simultaneously;
acquiring tracked Range of Motion (ROM) shapes including the mesh sequence;
implementing semantic segmentation by categorizing aspects of the mesh sequence into a class;
voxelizing the mesh sequence;
extracting a volume of a muscle segment from the voxelized mesh sequence;
deforming the muscle segment; and
generating a new pose with the muscle deformation.
11 . The method of claim 10 wherein deforming the muscle is performed by a physical simulator.
12 . The method of claim 10 wherein the volume is computed by determining edges of the muscle segment and then calculating the volume within the edges.
13 . An apparatus comprising:
a non-transitory memory for storing an application, the application for:
acquiring a rest pose including a mesh sequence including pose modeling and registration for blending, wherein the mesh sequence is acquired using an integrated photo-video volumetric capture system for 3D/4D scans by acquiring images and videos of a subject simultaneously;
acquiring tracked Range of Motion (ROM) shapes including the mesh sequence;
implementing semantic segmentation by categorizing aspects of the mesh sequence into a class;
voxelizing the mesh sequence;
extracting a volume of a muscle segment from the voxelized mesh sequence;
deforming the muscle segment; and
generating a new pose with the muscle deformation; and
a processor coupled to the memory, the processor configured for processing the application.
14 . The apparatus of claim 13 wherein deforming the muscle is performed by a physical simulator.
15 . The apparatus of claim 13 wherein the volume is computed by determining edges of the muscle segment and then calculating the volume within the edges.
16 . A system comprising:
a volumetric capture system for 3D and 4D scanning including capturing photos and video simultaneously, wherein the 3D scanning and 4D scanning includes detecting muscle deformation of an actor; and
a computing device configured for:
acquiring a rest pose including a mesh sequence including pose modeling and registration for blending, wherein the mesh sequence is acquired using an integrated photo-video volumetric capture system for 3D/4D scans by acquiring images and videos of a subject simultaneously;
acquiring tracked Range of Motion (ROM) shapes including the mesh sequence;
implementing semantic segmentation by categorizing aspects of the mesh sequence into a class;
voxelizing the mesh sequence;
extracting a volume of a muscle segment from the voxelized mesh sequence;
deforming the muscle segment; and
generating a new pose with the muscle deformation.
17 . The system of claim 16 wherein deforming the muscle is performed by a physical simulator.
18 . The system of claim 16 wherein the volume is computed by determining edges of the muscle segment and then calculating the volume within the edges.