Topologically consistent multi-view face inference using volumetric sampling
Systems and methods are provided for generating topologically consistent meshes across various subjects, objects, and/or various facial expressions using a volumetric representation. In one example, a progressive mesh generation network is configured to embed the topological structure of a subject or an object in a feature volume sampled from a geometry-aware local features. Further, a coarse-to-fine iterative architecture facilitates dense and accurate facial mesh predictions using a consistent mesh topology. In another example, one or more high-quality asset maps may be generated from a final base mesh.
1 . A system, comprising:
one or more processors;
one or more memories storing instructions that are executable by the one or more processors to cause the system to:
receive at least two input images of a subject or an object, each of the at least two input images corresponding to a different view of the subject or the object;
generate a three-dimensional initial coarse mesh of the subject or the object based on the at least two input images, the three-dimensional initial coarse mesh comprising a first number of vertices, wherein generating the three-dimensional coarse mesh comprises:
generating one or more two-dimensional feature maps for each input image;
generating a global feature volume within a global three-dimensional grid based on the one or more two-dimensional feature maps; and
inferring a corresponding vertex location for each of the number of vertices of the initial coarse mesh based on a probability distribution for the corresponding vertex location, wherein inferring the corresponding vertex locations comprises inputting the global feature volume into a global geometry network trained to output a plurality of channels, each channel encoding a corresponding probability distribution for the corresponding vertex location; and
generate a three-dimensional final base mesh of the subject or the object from the initial coarse mesh, the three-dimensional final base mesh comprising a final number of vertices greater than the first number of vertices of the initial coarse mesh;
wherein generating the three-dimensional base mesh from the initial coarse mesh comprises iteratively upsampling and refining a plurality of vertices, starting from the initial coarse mesh to generate a series of intermediate meshes, each successive intermediate mesh having a greater number of vertices than a previous intermediate mesh and less than the final number of vertices of the final base mesh, and wherein the upsampling is based on additional normal displacement weights.
2 . The system of claim 1 , wherein the feature volume is generated by bilinear sampling and fusing a plurality of two-dimensional image features from each of the two-dimensional feature maps on to corresponding projected coordinates within the three-dimensional grid.
3 . The system of claim 1 , wherein the global geometry network has a U-net architecture with skip connections.
4 . The system of claim 1 , wherein refining the plurality of vertices comprises updating corresponding positions of the plurality of vertices.
5 . The system of claim 1 , wherein refining the plurality of vertices comprises for each of the plurality of vertices, generating a local volumetric grid, generating a local feature volume for each grid, and updating the position of the corresponding vertex based on a second probability distribution for the position of the corresponding vertex, the second probability distribution based on the corresponding local feature volume.
6 . A system, comprising:
one or more processors;
one or more memories storing instructions that are executable by the one or more processors to cause the system to at least:
receive a plurality of images of a subject or an object, each the plurality of images corresponding to a different camera view angle and a plurality of corresponding camera calibration parameters for each image;
generate a three-dimensional coarse mesh based on the plurality of images and the plurality of camera calibration parameters, the three-dimensional initial coarse mesh comprising a first number of vertices;
generate a global feature volume comprising a plurality of global volumetric features within a global three-dimensional grid based on the plurality of images;
for each vertex of the initial coarse mesh, infer a corresponding probability distribution for a corresponding location of a corresponding vertex based on the global volumetric features, wherein the probability distribution for the corresponding location of the corresponding vertex is inferred by inputting the global feature volume into a global geometry network trained to output a plurality of channels, each channel encoding the corresponding probability distribution for the corresponding vertex location; and
generate a three-dimensional final base mesh of the subject or the object from the initial coarse mesh, the three-dimensional final base mesh comprising a final number of vertices greater than the first number of vertices of the initial coarse mesh;
wherein generating the three-dimensional base mesh from the initial coarse mesh comprises iteratively generating a series of intermediate meshes, each successive intermediate mesh having a greater number of vertices than a previous intermediate mesh and less than the final number of vertices of the final base mesh, wherein the iterative generation of intermediate meshes includes upsampling a plurality of vertices based on additional normal displacement.
7 . The system of claim 6 , wherein the upsampling of the iteratively generating the three-dimensional base mesh from the initial coarse mesh is performed to successively increase a number of vertices and further refining corresponding positions of each vertex.
8 . The system of claim 7 , wherein refining corresponding positions of each vertex comprises for each vertex, generating a local volumetric grid, generating a local feature volume for each grid, and updating the position of the corresponding vertex based on a second probability distribution for the position of the corresponding vertex, the second probability distribution based on the corresponding local feature volume.
9 . A method, comprising:
receiving, at one or more processors, a plurality of images of the subject or the object, each of the plurality of the images comprising a different view of the subject or the object;
generating, via the one or more processors, a three-dimensional initial coarse mesh based on the plurality of images according to a global volumetric feature sampling algorithm, the initial coarse mesh comprising a number of vertices and inferring a corresponding vertex location for each of the number of vertices of the initial coarse mesh based on a probability distribution for the corresponding vertex location, wherein inferring the corresponding vertex locations comprises inputting a global feature volume into a global geometry network trained to output a plurality of channels, each channel encoding a corresponding probability distribution for the corresponding vertex location;
generating, via the one or more processors, a three-dimensional final base mesh of the subject or the object;
generating, via the one or more processors, an image rendering of the subject or the object based on the three-dimensional final base mesh; and
displaying, via a display portion of a user interface communicatively coupled to the one or more processors, one or more of the initial coarse mesh, the final base mesh, and the image rendering;
wherein generating the three-dimensional final base mesh comprises iteratively upsampling starting from the initial coarse mesh to generate a series of intermediate meshes with increasing mesh density, and wherein the upsampling is based on additional normal displacement weights.
10 . The method of claim 9 , wherein generating the three-dimensional final base mesh further comprises after each upsampling, refining corresponding positions of vertices in each of the initial coarse mesh and each of the series of intermediate meshes.
11 . The method of claim 9 , wherein the global geometry network has a U-net architecture with skip connections.
12 . The method of claim 10 , wherein refining the corresponding positions of vertices comprises for each vertex, generating a local volumetric grid, generating a local feature volume for each grid, and updating the position of the corresponding vertex based on a second probability distribution for the position of the corresponding vertex, the second probability distribution based on the corresponding local feature volume.