Photometric-based 3D object modeling
Aspects of the present disclosure involve a system and a method for performing operations comprising: accessing a source image depicting a target structure; accessing one or more target images depicting at least a portion of the target structure; computing correspondence between a first set of pixels in the source image of a first portion of the target structure and a second set of pixels in the one or more target images of the first portion of the target structure, the correspondence being computed as a function of camera parameters that vary between the source image and the one or more target images; and generating a three-dimensional (3D) model of the target structure based on the correspondence between the first set of pixels in the source image and the second set of pixels in the one or more target images based on a joint optimization of target structure and camera parameters.
1 . A method comprising:
accessing a source image depicting a target structure;
accessing one or more target images depicting at least a portion of the target structure;
computing a first set of distances between each pixel in a first collection of pixels corresponding to the portion of the target structure depicted in the source image;
computing a second set of distance between each pixel in a second collection of pixels corresponding to the portion of the target structure depicted in the one or more target images;
selecting a sampling parameter based on at least one difference between the first and second sets of distances; and
generating a three-dimensional (3D) model of the target structure based on the selected sampling parameter.
2 . The method of claim 1 , further comprising:
generating the 3D model based on a joint optimization of the target structure and one or more camera parameters.
3 . The method of claim 2 , wherein the joint optimization comprises solving an optimization problem that is based on a cost function that relates pixels of the portion of the target structure in the source image to pixels in the one or more target images.
4 . The method of claim 1 , further comprising:
identifying the second collection of pixels as a function of a set of camera parameters; and
reducing photometric error to generate the 3D model.
5 . The method of claim 1 , further comprising:
un-distorting the first collection of pixels based on a set of camera parameters.
6 . The method of claim 1 , further comprising normalizing the first and second collection of pixels.
7 . The method of claim 1 , further comprising computing a sum of squares of computed differences between each pixel in the first and second collection of pixels.
8 . The method of claim 1 , further comprising:
computing a pixel to 3D coordinate correspondence between pixels in the source image and a 3D point on the target structure; and
computing a 3D coordinate to pixel correspondence between a 3D point on the target structure and a pixel in the one or more target images.
9 . The method of claim 1 , further comprising:
defining an optimization problem comprising a plurality of structure parameters and one or more camera parameters, the optimization problem being lighting invariant and surface normal invariant.
10 . The method of claim 9 , wherein solving the optimization problem comprises decoupling camera parameter updates from structure parameter updates thereby to reduce an amount of data that is stored.
11 . The method of claim 1 , wherein the source image and the one or more target images are received in real-time in a camera feed from a camera on a device.
12 . The method of claim 11 , further comprising:
accessing an augmented reality content item comprising an augmented reality effect; and
overlaying the augmented reality content item onto the camera feed based on the 3D model to provide an augmented reality experience in which the augmented reality content item is displayed as part of the camera feed.
13 . The method of claim 1 , wherein the source image and the one or more target images are previously captured and processed offline on a server.
14 . The method of claim 1 , further comprising:
adjusting a resolution of the source image to a resolution of the one or more target images.
15 . The method of claim 1 , further comprising up-sampling or down-sampling the one or more target images based on the sampling parameter.
16 . The method of claim 1 , further comprising:
generating a 3D coordinate frame of the target structure;
computing visibility of the 3D coordinate frame for a plurality of images as a depth map; and
selecting one of the plurality of images as the source image based on the computed visibility.
17 . The method of claim 16 , further comprising:
computing a grid of pixels having specified spacing corresponding to the 3D coordinate frame;
sampling the plurality of images associated with the grid of pixels to generate a matrix, each column of the matrix corresponding to a different one of the plurality of images;
computing a mean, a weighted mean, or a solution to a robustified sum of squares of the columns of the matrix; and
selecting as the source image an image of the plurality of images for which the corresponding column is closest in value to any one of the computed mean, the weighted mean, or the solution to the robustified sum of squares respectively.
18 . The method of claim 1 , further comprising:
processing a first set of images that are reduced in size during an initial phase of optimization; and
processing a second set of images as the one or more target images that are larger in size following the initial phase of optimization to improve convergence.
19 . A system comprising:
at least one processor configured to perform operations comprising:
accessing a source image depicting a target structure;
accessing one or more target images depicting at least a portion of the target structure;
computing a first set of distances between each pixel in a first collection of pixels corresponding to the portion of the target structure depicted in the source image;
computing a second set of distance between each pixel in a second collection of pixels corresponding to the portion of the target structure depicted in the one or more target images;
selecting a sampling parameter based on at least one difference between the first and second sets of distances; and
generating a three-dimensional (3D) model of the target structure based on the selected sampling parameter.
20 . A non-transitory machine-readable storage medium that includes instructions that, when executed by at least one processor of a machine, cause the machine to perform operations comprising:
accessing a source image depicting a target structure;
accessing one or more target images depicting at least a portion of the target structure;
computing a first set of distances between each pixel in a first collection of pixels corresponding to the portion of the target structure depicted in the source image;
computing a second set of distance between each pixel in a second collection of pixels corresponding to the portion of the target structure depicted in the one or more target images;
selecting a sampling parameter based on at least one difference between the first and second sets of distances; and
generating a three-dimensional (3D) model of the target structure based on the selected sampling parameter.