Single-pass object scanning
Various implementations disclosed herein include devices, systems, and methods that generates a three-dimensional (3D) model based on a selected subset of the images and depth data corresponding to each of the images of the subset. For example, an example process may include acquiring sensor data during movement of the device in a physical environment including an object, the sensor data including images of a physical environment captured via a camera on the device, selecting a subset of the images based on assessing the images with respect to motion-based defects based on device motion and depth data, and generating a 3D model of the object based on the selected subset of the images and depth data corresponding to each of the images of the selected subset.
1 . A method comprising:
at a device having a processor:
acquiring a plurality of images during movement of the device in a physical environment comprising an object, the plurality of images having center regions depicting portions of the physical environment;
determining distances from an object feature to the portions of the physical environment depicted at the center regions of the plurality of images; and
selecting a subset of the images based on the distances from the object feature to the portions of the physical environment depicted at the center regions of the plurality of images.
2 . The method of claim 1 , wherein selecting the subset of images comprises excluding images having distances from the object feature to the portions of the physical environment depicted at the center regions of the plurality of images that have distances greater than a threshold.
3 . The method of claim 1 , wherein determining the distances from the object feature to the portions of the physical environment depicted at the center regions of the plurality of images comprises determining distances to one or more edges of the object feature.
4 . The method of claim 1 , wherein determining the distances from the object feature to the portions of the physical environment depicted at the center regions of the plurality of images comprises determining the distances in a three-dimensional (3D) coordinate system corresponding to the physical environment.
5 . The method of claim 1 , wherein the subset of images is selected for use as key frames in generating a three-dimensional (3D) model of the object.
6 . The method of claim 1 , further comprising:
displaying the physical environment; and
providing a feedback mechanism comprising highlighting areas on the object in the displayed physical environment that have already been captured.
7 . The method of claim 1 , wherein selecting the subset of the images based on the distances from the object feature to the portions of the physical environment depicted at the center regions of the plurality of images comprises determining a projection speed of particular pixels associated with the object.
8 . The method of claim 7 , wherein the particular pixels are associated with pixels corresponding to an identification of the object, a 3D bounding box corresponding to the object, or a center of a view of a camera of the device.
9 . The method of claim 7 , wherein determining the projection speed of a pixel associated with the object includes determining a pixel's speed based on a current motion of a camera of the device and a distance of the pixel from the camera.
10 . The method of claim 1 , wherein selecting the subset of the images based on the distances from the object feature to the portions of the physical environment depicted at the center regions of the plurality of images is based on:
determining which segments of the subset of the images are within a distance range; and
updating the subset of the images to not include the segments of the images that are outside of the distance range.
11 . The method of claim 1 , wherein selecting the subset of the images based on the distances from the object feature or model of the object to the portions of the physical environment depicted at the center regions of the plurality of images is based on:
determining which segments of the subset of the images that are within an angle range; and
updating the subset of the images to not include the segments of depth data that are outside of the angle range.
12 . The method of claim 1 , wherein selecting the subset of the images based on the distances from the object feature or model of the object to the portions of the physical environment depicted at the center regions of the plurality of images comprises:
determining a preliminary object model corresponding to the object;
determining an edge of the preliminary object model;
determining which segments of the subset of the images include a distance of a center of view of a camera of the device to the edge of the preliminary object model that are within an object model distance threshold; and
updating the subset of the images to not include segments of the subset of the images that are outside of the object model distance threshold.
13 . The method of claim 1 , wherein selecting the subset of the images based on the distances from the object feature or model of the object to the portions of the physical environment depicted at the center regions of the plurality of images comprises:
selecting segments of the subset of the images that includes the object or particular features of the object.
14 . The method of claim 1 , wherein the device includes a user interface, wherein, during the movement of the device, the user interface displays:
the physical environment including the object; and
a preliminary 3D model of the object based on depth data associated with the plurality of images.
15 . The method of claim 14 , wherein the preliminary 3D model is generated during the movement of the device based on the selected subset of the images, and the preliminary 3D model is displayed simultaneously with images of the physical environment captured via a camera on the device.
16 . The method of claim 1 , wherein the object feature is an edge of a preliminary object model having a position relative to the physical environment based on a position of the object in the physical environment.
17 . A device comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising:
acquiring a plurality of images during movement of the device in a physical environment comprising an object, the plurality of images having center regions depicting portions of the physical environment;
determining distances from an object feature to the portions of the physical environment depicted at the center regions of the plurality of images; and
selecting a subset of the images based on the distances from the object feature to the portions of the physical environment depicted at the center regions of the plurality of images.
18 . The device of claim 17 , wherein selecting the subset of images comprises excluding images having distances from the object feature to the portions of the physical environment depicted at the center regions of the plurality of images that have distances greater than a threshold.
19 . The device of claim 17 , wherein determining the distances from the object feature to the portions of the physical environment depicted at the center regions of the plurality of images comprises determining distances to one or more edges of the object feature.
20 . The device of claim 17 , wherein determining the distances from the object feature to the portions of the physical environment depicted at the center regions of the plurality of images comprises determining the distances in a three-dimensional (3D) coordinate system corresponding to the physical environment.
21 . A non-transitory computer-readable storage medium, storing computer-executable program instructions on a computer to perform operations comprising:
acquiring a plurality of images during movement of a device in a physical environment comprising an object, the plurality of images having center regions depicting portions of the physical environment;
determining distances from an object feature to the portions of the physical environment depicted at the center regions of the plurality of images; and
selecting a subset of the images based on the distances from the object feature to the portions of the physical environment depicted at the center regions of the plurality of images.