Spatial scanning for extended reality
An extended Reality (XR) system that provides services for determining 3D data of physical objects in a real-world scene. The XR system receives a request from an application to initiate a spatial scan of a real-world scene. In response, the XR system captures video frame data of the real-world scene and captures a pose of the XR system. The XR system determines a physical object in the real-world scene and determines a 2D position of the physical object, using the video frame data. The XR system determines a depth of the physical object using the 2D position and determines a 3D position of the physical object in the real-world scene using the 2D position of the physical object, the depth of the physical object, and the pose of the XR system. The XR system communicates the 3D position data to the application.
1. A computer-implemented method, comprising:
receiving, by one or more processors of an extended Reality (XR) system, a request to initiate a spatial scan of a real-world scene;
capturing, by the one or more processors, using one or more cameras of the XR system, video frame data of the real-world scene;
capturing, by the one or more processors, using a pose tracking component of the XR system, a pose of the XR system;
detecting, by the one or more processors, a physical object in the real-world scene and determining a two-dimensional (2D) position of the physical object, using the video frame data;
determining, by the one or more processors, a depth of the physical object using the 2D position; and
determining, by the one or more processors, a three-dimensional (3D) position of the physical object in the real-world scene using the 2D position of the physical object, the depth of the physical object, and the pose of the XR system.
2. The computer-implemented method of claim 1 , further comprising communicating the 3D position of the physical object to an application of the XR system.
3. The computer-implemented method of claim 2 , further comprising generating a label for the physical object.
4. The computer-implemented method of claim 1 , wherein detecting the physical object and determining the 2D position of the physical object comprises detecting the physical object based on artificial intelligence methodologies using an object identification model.
5. The computer-implemented method of claim 1 , wherein determining the 3D position of the physical object comprises:
generating a geometric ray using a point of view of a user using the XR system and the 2D position of the physical object; and
determining the 3D position of the physical object using an intersection between the geometric ray and the physical object.
6. The computer-implemented method of claim 1 , wherein the XR system comprises a head-wearable apparatus.
7. The computer-implemented method of claim 1 , wherein the XR system comprises a mobile device.
8. A machine comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
receiving a request to initiate a spatial scan of a real-world scene;
capturing using one or more cameras of an extended Reality (XR) system, video frame data of the real-world scene;
capturing, using a pose tracking component of the XR system, a pose of the XR system;
detecting a physical object in the real-world scene and determining a 2D position of the physical object, using the video frame data;
determining a depth of the physical object using the 2D position; and
determining a 3D position of the physical object in the real-world scene using the 2D position of the physical object, the depth of the physical object, and the pose of the XR system.
9. The machine of claim 8 , wherein the operations further comprise communicating the 3D position of the physical object to an application of the XR system.
10. The machine of claim 9 , wherein the operations further comprise generating a label for the physical object.
11. The machine of claim 8 , wherein detecting the physical object and determining the 2D position of the physical object comprises detecting the physical object based on artificial intelligence methodologies using an object identification model.
12. The machine of claim 8 , wherein determining the 3D position of the physical object comprises:
generating a geometric ray using a point of view of a user using the XR system and the 2D position of the physical object; and
determining the 3D position of the physical object using an intersection between the geometric ray and the physical object.
13. The machine of claim 8 , wherein the XR system comprises a head-wearable apparatus.
14. The machine of claim 8 , wherein the XR system comprises a mobile device.
15. A non-transitory machine-readable storage medium, the machine-readable storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:
receiving a request to initiate a spatial scan of a real-world scene;
capturing using one or more cameras of an extended Reality (XR) system, video frame data of the real-world scene;
capturing, using a pose tracking component of the XR system, a pose of the XR system;
detecting a physical object in the real-world scene and determining a 2D position of the physical object, using the video frame data;
determining a depth of the physical object using the 2D position; and
determining a 3D position of the physical object in the real-world scene using the 2D position of the physical object, the depth of the physical object, and the pose of the XR system.
16. The non-transitory machine-readable storage medium of claim 15 , wherein the operations further comprise communicating the 3D position of the physical object to an application of the XR system.
17. The non-transitory machine-readable storage medium of claim 16 , wherein the operations further comprise generating a label for the physical object.
18. The non-transitory machine-readable storage medium of claim 15 , wherein detecting the physical object and determining the 2D position of the physical object comprises detecting the physical object based on artificial intelligence methodologies using an object identification model.
19. The non-transitory machine-readable storage medium of claim 15 , wherein determining the 3D position of the physical object comprises:
generating a geometric ray using a point of view of a user using the XR system and the 2D position of the physical object; and
determining the 3D position of the physical object using an intersection between the geometric ray and the physical object.
20. The non-transitory machine-readable storage medium of claim 15 , wherein the XR system comprises a head-wearable apparatus.