Automatic retopologization of textured 3D meshes
Methods and apparatuses for automating the retopologization of 3D meshes including the automated selection and adjustment of correspondence points are described. The automated selection of correspondence points may be performed to refine locations of correspondence points using a matching score that is computed based on surface normal similarity between surfaces corresponding with a candidate correspondence point on an input scan mesh and a point on a morphable model of 3D surfaces. The matching score may also take into account a distance between a candidate correspondence point on the input scan mesh and a corresponding point on the morphable model of 3D surfaces and similarities in surface features, such as similarities in surface curvature at the candidate correspondence point on the input scan mesh and the corresponding point on the morphable model of 3D surfaces.
1 . A system for automating the retopologization of 3D meshes, comprising:
a storage device for storing instructions that, when executed, cause the system to perform operations comprising:
generating an input scan mesh associated with an object;
generating an intermediate mesh associated with the object that has a geometry that approximates the input scan mesh;
identifying a keypoint on the intermediate mesh;
determining a threshold distance based on a degree of uncertainty associated with the keypoint on the intermediate mesh;
determining a first surface normal associated with the keypoint;
displacing the keypoint in the direction of the first surface normal by the threshold distance;
calculating a matching score based on a distance between the keypoint and a candidate correspondence point on the input scan mesh;
selecting the candidate correspondence point based on the matching score; and
adjusting the intermediate mesh using the candidate correspondence point.
2 . The system of claim 1 , further comprising instructions that, when executed, cause the system to perform operations comprising:
adjusting the keypoint to the candidate correspondence point; and
warping the intermediate mesh using the candidate correspondence point.
3 . The system of claim 1 , further comprising instructions that, when executed, cause the system to perform operations comprising:
generating a 3D mesh for the object by applying photometric refinement to the intermediate mesh.
4 . The system of claim 1 , further comprising instructions that, when executed, cause the system to perform operations comprising:
determining a surface curvature associated with a first surface at the keypoint; and
calculating the matching score based on the surface curvature associated with the first surface at the keypoint.
5 . The system of claim 1 , further comprising instructions that, when executed, cause the system to perform operations comprising:
determining a vector difference between the first surface normal and a second surface normal associated with the candidate correspondence point; and
calculating the matching score based on the vector difference.
6 . The system of claim 1 , further comprising instructions that, when executed, cause the system to perform operations comprising:
acquiring a set of images that capture different views of the object; and
generating the input scan mesh using the set of images.
7 . The system of claim 1 , further comprising instructions that, when executed, cause the system to perform operations comprising:
restricting the candidate correspondence point to be within the threshold distance of the keypoint.
8 . The system of claim 1 , wherein:
the object comprises a human head; and
the intermediate mesh has fewer surfaces than the input scan mesh.
9 . The system of claim 1 , wherein:
the intermediate mesh comprises a morphable model of 3D surfaces that is fit to the input scan mesh using keypoint regression techniques.
10 . The system of claim 1 , further comprising instructions that, when executed, cause the system to perform operations comprising:
generating the intermediate mesh via keypoint regression using a deep neural network; and
identifying the candidate correspondence point out of a set of correspondence points with the highest matching score.
11 . A method for automating the retopologization of 3D meshes, comprising:
generating an input scan mesh associated with an object;
generating an intermediate mesh associated with the object that has a geometry that approximates the input scan mesh;
identifying a keypoint on the intermediate mesh;
determining a threshold distance based on a degree of certainty associated with the keypoint;
determining a first surface normal associated with the keypoint and a second surface normal associated with the candidate correspondence point;
displacing the keypoint in the direction of the first surface normal by the threshold distance;
calculating a matching score based on a distance between the keypoint and a candidate correspondence point on the input scan mesh, the first surface normal, and the second surface normal;
selecting the candidate correspondence point based on the matching score; and
adjusting the intermediate mesh using the candidate correspondence point.
12 . The method of claim 11 , wherein:
the adjusting the intermediate mesh includes warping the intermediate mesh based on a position of the candidate correspondence point.
13 . The method of claim 11 , further comprising:
generating a 3D mesh for the object by applying photometric refinement techniques to the intermediate mesh.
14 . The method of claim 11 , further comprising:
determining a surface curvature associated with a first surface at the keypoint; and
calculating the matching score based on the surface curvature associated with the first surface at the keypoint.
15 . The method of claim 11 , further comprising:
determining a vector difference between the first surface normal and the second surface normal; and
calculating the matching score based on the vector difference.
16 . The method of claim 11 , further comprising:
acquiring a set of images that capture different views of the object; and
generating the input scan mesh using the set of images.
17 . The method of claim 11 , wherein:
the object comprises a human torso; and
the intermediate mesh has fewer surfaces than the input scan mesh.
18 . The method of claim 11 , wherein:
the intermediate mesh comprises a morphable model of 3D surfaces that is fit to the input scan mesh using keypoint regression techniques.
19 . The method of claim 11 , further comprising:
identifying the candidate correspondence point out of a set of correspondence points with the highest matching score.
20 . A system for automating the retopologization of 3D meshes, comprising:
a storage device configured to store a source 3D mesh; and
a processing system in communication with the storage device that is configured to:
generate a target 3D mesh that has a geometry that approximates the source 3D mesh;
identify a keypoint on the target 3D mesh;
determine a threshold distance based on a degree of certainty associated with the keypoint;
determine a first surface normal associated with the keypoint and a second surface normal associated with the candidate correspondence point;
determine a difference in surface curvature for a first surface associated with the keypoint and a second surface associated with the candidate correspondence point;
displace the keypoint in the direction of the first surface normal by the threshold distance;
calculate a matching score based on a distance between the keypoint and a candidate correspondence point on the source 3D mesh, a difference between the first surface normal and the second surface normal, and the difference in surface curvature;
select the candidate correspondence point based on the matching score;
warp the target 3D mesh using the candidate correspondence point; and
apply photometric refinement to the target 3D mesh subsequent to warping the target 3D mesh.