Object detection using multiple three dimensional scans
One exemplary implementation facilitates object detection using multiple scans of an object in different conditions. For example, a first scan of the object can be created by capturing images of the object by moving an image sensor on a first path in a first condition, e.g., bright lighting. A second scan of the object can then be created by capturing additional images of the object by moving the image sensor on a second path in a second condition, e.g., dim lighting. Implementations determine a transform that associates the scan data from these multiple scans to one another and use the transforms to generate a 3D model of the object in a single coordinate system. Augmented content can be positioned relative to that object in the single coordinate system and thus will be displayed in the appropriate location regardless of the condition in which the physical object is later detected.
1. A method comprising:
at a device comprising a processor, a non-transitory computer-readable storage medium, and an image sensor:
obtaining first scan data of a physical object in a first condition using the image sensor, the first scan data comprising images from a plurality of image sensor positions defined in a first coordinate system;
obtaining second scan data of the physical object in a second condition using the image sensor, the second condition different from the first condition, the second scan data comprising images from a plurality of image sensor positions defined in a second coordinate system;
determining, via the processor, a transform between the first coordinate system and the second coordinate system, wherein determining the transform comprises determining an offset between poses of the image sensor associated with a first image of the first scan data and a second image of the second scan data; and
generating a three dimensional (3D) model of the physical object based on the first scan data, the second scan data, and the transform.
2. The method of claim 1 , wherein determining the transform comprises matching the first image of the first scan data with the second image of the second scan data.
3. The method of claim 1 , wherein determining the transform comprises matching multiple images of the first scan data with multiple images of the second scan data.
4. The method of claim 1 , wherein determining the transform comprises aligning a first point cloud associated with the first scan data with a second point cloud associated with the second scan data.
5. The method of claim 1 , wherein the 3D model is a point cloud of points associated with descriptors.
6. The method of claim 5 , wherein the point cloud includes points having descriptors based on the first scan data and points having descriptors based on the second scan data.
7. The method of claim 5 , wherein generating the 3D model comprises merging a first point having a descriptor based on the first scan data with a second point having a descriptor based on the second scan data.
8. The method of claim 7 further comprising determining to merge the first point and the second point based on proximity of the first point to the second point.
9. The method of claim 1 , wherein generating the 3D model comprises representing the first scan data and the second scan data in a single coordinate system.
10. The method of claim 9 further comprising associating augmented content with the 3D model at a position relative to the single coordinate system.
11. The method of claim 1 further comprising:
obtaining image data via the image sensor;
detecting the physical object using the 3D model and the image data; and
aligning the 3D model with the physical object using a single coordinate system.
12. The method of claim 11 , wherein detecting the physical object comprises matching the image data with point cloud descriptors of the 3D model.
13. The method of claim 11 , wherein detecting the physical object comprises determining a current pose of the image sensor relative to the 3D model.
14. The method of claim 11 further comprising displaying a computer generated reality (CGR) environment depicting the physical object based on the image data and augmented content, wherein the augmented content is positioned based on aligning the 3D model with the physical object using the single coordinate system.
15. The method of claim 1 , wherein the first and second conditions are different lighting conditions.
16. The method of claim 1 , wherein the first and second conditions comprise different object states of an object in the physical object.
17. The method of claim 1 further comprising determining to request a second scan based on testing the first scan data.
18. A system comprising:
a non-transitory computer-readable storage medium;
a image sensor; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
obtaining first scan data of a physical object in a first condition using the image sensor, the first scan data comprising images from a plurality of image sensor positions defined in a first coordinate system;
obtaining second scan data of the physical object in a second condition using the image sensor, the second condition different from the first condition, the second scan data comprising images from a plurality of image sensor positions defined in a second coordinate system;
determining, via the processor, a transform between the first coordinate system and the second coordinate system, wherein determining the transform comprises determining an offset between poses of the image sensor associated with a first image of the first scan data and a second image of the second scan data; and
generating a three dimensional (3D) model of the physical object based on the first scan data, the second scan data, and the transform.
19. A non-transitory computer-readable storage medium, storing program instructions computer-executable on a computer to perform operations comprising:
obtaining first scan data of a physical object in a first condition using an image sensor, the first scan data comprising images from a plurality of image sensor positions defined in a first coordinate system;
obtaining second scan data of the physical object in a second condition using the image sensor, the second condition different from the first condition, the second scan data comprising images from a plurality of image sensor positions defined in a second coordinate system;
determining, via a processor, a transform between the first coordinate system and the second coordinate system, wherein determining the transform comprises determining an offset between poses of the image sensor associated with a first image of the first scan data and a second image of the second scan data; and
generating a three dimensional (3D) model of the physical object based on the first scan data, the second scan data, and the transform.