Generating enhanced three-dimensional object reconstruction models from sparse set of object images
Enhanced methods and systems for generating both a geometry model and an optical-reflectance model (an object reconstruction model) for a physical object, based on a sparse set of images of the object under a sparse set of viewpoints. The geometry model is a mesh model that includes a set of vertices representing the object's surface. The reflectance model is SVBRDF that is parameterized via multiple channels (e.g., diffuse albedo, surface-roughness, specular albedo, and surface-normals). For each vertex of the geometry model, the reflectance model includes a value for each of the multiple channels. The object reconstruction model is employed to render graphical representations of a virtualized object (a VO based on the physical object) within a computation-based (e.g., a virtual or immersive) environment. Via the reconstruction model, the VO may be rendered from arbitrary viewpoints and under arbitrary lighting conditions.
1. A method for a reconstruction of a three-dimensional ( 3 D) physical object, the method comprising:
receiving a set of images, wherein each image of the set of images depicts the physical object from a separate viewpoint of a set of viewpoints;
generating, based on the set of image, a set of per-view (PV) reflectance maps, a set of composite feature maps, and a coarse geometry model encoding a coarse representation of a geometry of a surface of the physical object, wherein each PV reflectance map of the set of PV reflectance maps corresponds to one of the viewpoints of the set of viewpoints and each composite feature map of the set of composite feature maps corresponds to one of the viewpoints of the set of viewpoints and is based a composite of the set of images warped to the corresponding viewpoint;
aggregating the set of PV reflectance maps across the set of viewpoints to generate a coarse reflectance model that includes a set of estimated reflectance parameters for each vertex of the coarse geometry model and for each viewpoint of the set of viewpoints; and
jointly refining, based on the set of composite feature maps, the coarse geometry model and the coarse reflectance model to generate a refined geometry model and a refined reflectance model, wherein the refined geometry model encodes a refined representation of the geometry and the refined reflectance models encodes a refined representation of the reflectance property, wherein the coarse geometry model and the coarse reflectance model are refined by fusing the set of estimated reflectance parameters for each vertex to reconstruct geometry and reflectance associated with the physical object.
2. The method for claim 1 , further comprising:
generating a set of PV depth maps based on the set of images, wherein each PV depth map of the set of PV depth maps corresponds to one of the viewpoints of the set of viewpoints;
aggregating the set of PV depth maps across the set of viewpoints to generate the set of PV reflectance maps and the set of composite feature maps; and
employing the set of PV depth maps to generate the coarse geometry model.
3. The method of claim 2 , further comprising:
generating a set of PV feature maps based on the set of images, wherein each PV feature map of the set of PV feature maps corresponds to one of the viewpoints of the set of viewpoints;
for each viewpoint of the set of viewpoints, warping each PV feature map to each of the other viewpoints of the set of viewpoints to generate a set of warped feature maps; and
generating the set of PV depth maps based on the set of warped feature maps.
4. The method of claim 2 , further comprising:
generating a set of PV surface-normal maps based on the set of PV depth maps, wherein each PV surface-normal map of the set of PV surface-normal maps corresponds to one of the viewpoints of the set of viewpoints;
generating a set of PV diffuse albedo maps based on the set of PV depth maps, wherein each PV diffuse albedo map of the set of PV diffuse albedo maps corresponds to one of the viewpoints of the set of viewpoints;
generating a set of PV surface-roughness maps based on the set of PV depth maps, wherein each PV surface-roughness map of the set of PV surface-roughness maps corresponds to one of the viewpoints of the set of viewpoints;
generating a set of PV specular albedo maps based on the set of PV depth maps, wherein each PV specular map of the set of PV specular maps corresponds to one of the viewpoints of the set of viewpoints; and
generating the set of PV reflectance maps based on a combination of the set of PV surface-normal maps, the set of PV diffuse albedo maps, the set of PV surface-roughness maps, and the set of PV specular albedo maps.
5. The method of claim 2 , further comprising:
generating an intermediate feature map for each pair of viewpoints of the set of viewpoints; and
generating each composite feature map of the set of composite feature maps by aggregating the intermediate feature maps for each pair of viewpoints that includes a viewpoint of the pair of viewpoints.
6. The method of claim 1 , further comprising:
generating a synthetic image for each image of the set of input images based on the coarse geometry model, the coarse reflectance model, and the set of composite feature maps; and
determining an error function based on a comparison of each of the synthetic images with the corresponding image of the set of images; and
refining each of the coarse geometry model and the coarse reflectance model based on the error function.
7. A computing system for a reconstruction of a three-dimensional (3D) physical object, comprising:
a processor device; and
a computer-readable storage medium, coupled with the processor device, having instructions stored thereon, which, when executed by the processor device, perform actions comprising:
steps for generating a set of per-view (PV) reflectance maps based on a set of images, wherein each image of the set of images depicts the physical object from a separate viewpoint of a set of viewpoints and each PV reflectance map of the set of PV reflectance maps corresponds to one of the viewpoints of the set of viewpoints;
steps for generating a set of composite feature maps, wherein each composite feature map of the set of composite feature maps corresponds to one of the viewpoints of the set of viewpoints and is based on a composite of the set of images warped to the corresponding viewpoint;
steps for jointly refining, based on the set of composite feature maps, a coarse geometry model and a coarse reflectance model to generate a refined geometry model and a refined reflectance model, wherein the refined geometry model encodes a refined representation of the geometry and the refined reflectance models encodes a refined representation of the reflectance property, and wherein the coarse geometry model and the coarse reflectance model are refined by fusing a set of estimated reflectance parameters for each vertex to reconstruct geometry and reflectance.
8. The computing system of claim 7 , wherein the actions further comprise:
steps for generating a set of PV depth maps based on the set of images, wherein each PV depth map of the set of PV depth maps corresponds to one of the viewpoints of the set of viewpoints;
steps for aggregating the set of PV depth maps across the set of viewpoints to generate the set of PV reflectance maps and the set of composite feature maps; and
steps for employing the set of PV depth maps to generate the coarse geometry model.
9. The computing system of claim 8 , wherein the actions further comprise:
steps for generating a set of PV feature maps based on the set of images, wherein each PV feature maps of the set of PV feature maps corresponds to one of the viewpoints of the set of viewpoints;
for each viewpoint of the set of viewpoints, steps for warping each PV feature map to each of the other viewpoints of the set of viewpoints to generate a set of warped feature maps; and
steps for generating the set of PV depth maps based on the set of warped feature maps.
10. The computing system of claim 8 , wherein the actions further comprise:
steps for generating a set of PV surface-normal maps based on the set of PV depth maps, wherein each PV surface-normal map of the set of PV surface-normal maps corresponds to one of the viewpoints of the set of viewpoints;
steps for generating a set of PV diffuse albedo maps based on the set of PV depth maps, wherein each PV diffuse albedo map of the set of PV diffuse albedo maps corresponds to one of the viewpoints of the set of viewpoints;
steps for generating a set of PV surface-roughness maps based on the set of PV depth maps, wherein each PV surface-roughness map of the set of PV surface-roughness maps corresponds to one of the viewpoints of the set of viewpoints;
steps for generating a set of PV specular albedo maps based on the set of PV depth maps, wherein each PV specular map of the set of PV specular maps corresponds to one of the viewpoints of the set of viewpoints; and
steps for generating the set of PV reflectance maps based on a combination of the set of PV surface-normal maps, the set of PV diffuse albedo maps, the set of PV surface-roughness maps, and the set of PV specular albedo maps.
11. The computing system of claim 7 , wherein the actions further comprise:
steps for generating an intermediate feature map for each pair of viewpoints of the set of viewpoints; and
steps for generating each composite feature map of the set of composite feature maps by aggregating the intermediate feature maps for each pair of viewpoints that includes a viewpoint of the pair of viewpoints.
12. The computing system of claim 7 , wherein the actions further comprise:
steps for generating a synthetic image for each image of the set of input images based on the coarse geometry model, the coarse reflectance model, and the set of composite feature maps; and
steps for determining an error function based on a comparison of each of the synthetic images with the corresponding image of the set of images; and
steps for refining each of the coarse geometry model and the coarse reflectance model based on the error function.
13. The computing system of claim 7 , wherein the actions further comprise:
steps for generating a set of PV depth maps based on the set of images, wherein each PV depth map of the set of PV depth maps corresponds to one of the viewpoints of the set of viewpoints;
steps for generating a set of PV surface-normal maps based on the set of PV depth maps, wherein each PV surface-normal map of the set of PV surface-normal maps corresponds to one of the viewpoints of the set of viewpoints;
steps for employing the set of PV depth maps to generate a point cloud, wherein the point cloud includes a set of 3D points that approximates the geometry of the surface of the physical object;
for each point of the set of 3D points, steps for determining a normal vector that is substantially orthogonal to the surface of the physical object; and
steps for employing a Poisson reconstruction process to generated the coarse geometry model based on the set of 3D points and the normal vector for each point in the set of 3D points.
14. Anon-transitory computer-readable storage medium having instructions stored thereon for a reconstruction of a three-dimensional (3D) physical object, which, when executed by a processor of a computing device cause the computing device to perform actions comprising:
generating a set of per-view (PV) reflectance maps based on a set of images, wherein each image of the set of images depicts the physical object from a separate viewpoint of a set of viewpoints and each PV reflectance map of the set of PV reflectance maps corresponds to one of the viewpoints of the set of viewpoints;
generating a set of composite feature maps, wherein each composite feature map of the set of composite feature maps corresponds to one of the viewpoints of the set of viewpoints and is based on a composite of the set of images warped to the corresponding viewpoint;
jointly refining, based on the set of composite feature maps, a coarse geometry model and a coarse reflectance model to generate a refined geometry model and a refined reflectance model, wherein the refined geometry model encodes a refined representation of the geometry and the refined reflectance models encodes a refined representation of the reflectance property, and wherein the coarse geometry model and the coarse reflectance model are refined by fusing a set of estimated reflectance parameters for each vertex to reconstruct geometry and reflectance.
15. The non-transitory computer-readable storage medium of claim 14 , wherein the actions further comprise:
generating a set of PV depth maps based on the set of images, wherein each PV depth map of the set of PV depth maps corresponds to one of the viewpoints of the set of viewpoints;
aggregating the set of PV depth maps across the set of viewpoints to generate the set of PV reflectance maps and the set of composite feature maps; and
employing the set of PV depth maps to generate the coarse geometry model.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the actions further comprise:
generating a set of PV feature maps based on the set of images, wherein each PV feature maps of the set of PV feature maps corresponds to one of the viewpoints of the set of viewpoints;
for each viewpoint of the set of viewpoints, warping each PV feature map to each of the other viewpoints of the set of viewpoints to generate a set of warped feature maps; and
generating the set of PV depth maps based on the set of warped feature maps.
17. The non-transitory computer-readable storage medium of claim 15 , wherein the actions further comprise:
generating a set of PV surface-normal maps based on the set of PV depth maps, wherein each PV surface-normal map of the set of PV surface-normal maps corresponds to one of the viewpoints of the set of viewpoints;
generating a set of PV diffuse albedo maps based on the set of PV depth maps, wherein each PV diffuse albedo map of the set of PV diffuse albedo maps corresponds to one of the viewpoints of the set of viewpoints;
generating a set of PV surface-roughness maps based on the set of PV depth maps, wherein each PV surface-roughness map of the set of PV surface-roughness maps corresponds to one of the viewpoints of the set of viewpoints;
generating a set of PV specular albedo maps based on the set of PV depth maps, wherein each PV specular map of the set of PV specular maps corresponds to one of the viewpoints of the set of viewpoints; and
generating the set of PV reflectance maps based on a combination of the set of PV surface-normal maps, the set of PV diffuse albedo maps, the set of PV surface-roughness maps, and the set of PV specular albedo maps.
18. The non-transitory computer-readable storage medium of claim 14 , wherein the actions further comprise:
generating an intermediate feature map for each pair of viewpoints of the set of viewpoints; and
generating each composite feature map of the set of composite feature maps by aggregating the intermediate feature maps for each pair of viewpoints that includes a viewpoint of the pair of viewpoints.
19. The non-transitory computer-readable storage medium of claim 14 , wherein the actions further comprise:
generating a synthetic image for each image of the set of input images based on the coarse geometry model, the coarse reflectance model, and the set of composite feature maps;
determining an error function based on a comparison of each of the synthetic images with the corresponding image of the set of images; and
refining each of the coarse geometry model and the coarse reflectance model based on the error function.
20. The non-transitory computer-readable storage medium of claim 14 , wherein the actions further comprise:
generating a set of PV depth maps based on the set of images, wherein each PV depth map of the set of PV depth maps corresponds to one of the viewpoints of the set of viewpoints;
generating a set of PV surface-normal maps based on the set of PV depth maps, wherein each PV surface-normal map of the set of PV surface-normal maps corresponds to one of the viewpoints of the set of viewpoints;
employing the set of PV depth maps to generate a point cloud, wherein the point cloud includes a set of 3D points that approximates the geometry of the surface of the physical object;
for each point of the set of 3D points, determining a normal vector that is substantially orthogonal to the surface of the physical object; and
employing a Poisson reconstruction process to generated the coarse geometry model based on the set of 3D points and the normal vector for each point in the set of 3D points.