Remote object detection and local tracking using visual odometry
A system and method for offloading object detection are described. A server receives first sensor data from a first sensor of an augmented reality (AR) display device. The first sensor data indicates a pose of the AR display device relative to a first reference coordinate system. The server detects a physical object using second sensor data received from a second sensor of the AR display device. The server determines, based on the second sensor data, a pose of the physical object relative to the AR display device. The server then determines the pose of the physical object relative to the first reference coordinate system based on the pose of the physical object relative to the AR display device and the pose of the AR display device relative to the first reference coordinate system.
1. A method comprising:
receiving, by a server system, first sensor data from a first sensor of an augmented reality display device, the first sensor data indicating a pose of the augmented reality display device relative to a first reference coordinate system;
receiving, by the server system, second sensor data from a second sensor of the augmented reality display device;
detecting, by the server system, a physical object using the second sensor data;
determining, based on the second sensor data, a pose of the physical object relative to the augmented reality display device;
determining the pose of the physical object relative to the first reference coordinate system based on the pose of the physical object relative to the augmented reality display device and the pose of the augmented reality display device relative to the first reference coordinate system; and
providing the pose of the physical object relative to the first reference coordinate system to the augmented reality display device.
2. The method of claim 1 , wherein the augmented reality display device is configured to render a virtual object corresponding to the physical object, and display the virtual object in a display of the augmented reality display device using the pose of the physical object relative to the first reference coordinate system.
3. The method of claim 2 , wherein the virtual object is displayed in the display of the augmented reality display device independently from the pose of the physical object relative to the augmented reality display device.
4. The method of claim 1 , further comprising:
receiving, by the server system, third sensor data from a third sensor external to the augmented reality display device, the third sensor data indicating the pose of the augmented reality display device relative to the first reference coordinate system,
wherein determining the pose of the physical object relative to the first reference coordinate system is based on the pose of the physical object relative to the augmented reality display device and, using the third sensor data, the pose of the augmented reality display device relative to the first reference coordinate system.
5. The method of claim 4 , further comprising:
rendering a virtual object based on the pose of the physical object relative to the first reference coordinate system; and
providing the rendered virtual object to the augmented reality display device, wherein the augmented reality display device is configured to display the rendered virtual object based on the pose of the physical object relative to the first reference coordinate system.
6. The method of claim 1 , further comprising:
rendering a virtual object based on the pose of the physical object relative to the first reference coordinate system; and
providing the rendered virtual object to the augmented reality display device, wherein the augmented reality display device is configured to display the rendered virtual object based on the pose of the physical object relative to the first reference coordinate system.
7. The method of claim 1 , wherein the first sensor includes a visual odometry system configured to identify a position and an orientation of the augmented reality display device relative to the first reference coordinate system.
8. The method of claim 1 , wherein the second sensor includes an optical sensor configured to generate second sensor data based on an image of the physical object captured by the optical sensor.
9. The method of claim 1 , further comprising:
receiving, by the server system, a request from the augmented reality display device to offload a tracking process of the augmented reality display device to the server system;
in response to receiving the request, generating offloaded processed data based on the first sensor data and second sensor data; and
providing the offloaded processed data to the augmented reality display device, the augmented reality display device configured to present, in a display of the augmented reality display device, virtual objects based on the offloaded processed data.
10. The method of claim 9 , wherein third sensor data includes externally tracked data based on a location and an orientation of the AR display device relative to a third sensor that is external to the augmented reality display device, wherein the offloaded processed data is based on a combination of the first, second, and third sensor data.
11. A server system comprising:
one or more computer processors; and
one or more non-transitory computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the server system to perform operations comprising:
receiving first sensor data from a first sensor of an augmented reality display device, the first sensor data indicating a pose of the augmented reality display device relative to a first reference coordinate system;
receiving second sensor data from a second sensor of the augmented reality display device;
detecting a physical object using the second sensor data;
determining, based on the second sensor data, a pose of the physical object relative to the augmented reality display device;
determining the pose of the physical object relative to the first reference coordinate system based on the pose of the physical object relative to the augmented reality display device and the pose of the augmented reality display device relative to the first reference coordinate system; and
providing the pose of the physical object relative to the first reference coordinate system to the augmented reality display device.
12. The server system of claim 11 , wherein the augmented reality display device is configured to render a virtual object corresponding to the physical object, and display the virtual object in a display of the augmented reality display device using the pose of the physical object relative to the first reference coordinate system.
13. The server system of claim 12 , wherein the virtual object is displayed in the display of the augmented reality display device independently from the pose of the physical object relative to the augmented reality display device.
14. The server system of claim 11 , wherein the operations further comprise:
receiving third sensor data from a third sensor external to the augmented reality display device, the third sensor data indicating the pose of the augmented reality display device relative to the first reference coordinate system,
wherein determining the pose of the physical object relative to the first reference coordinate system is based on the pose of the physical object relative to the augmented reality display device and, using the third sensor data, the pose of the augmented reality display device relative to the first reference coordinate system.
15. The server system of claim 14 , wherein the operations further comprise:
rendering a virtual object based on the pose of the physical object relative to the first reference coordinate system; and
providing the rendered virtual object to the augmented reality display device, wherein the augmented reality display device is configured to display the rendered virtual object based on the pose of the physical object relative to the first reference coordinate system.
16. The server system of claim 11 , wherein the operations further comprise:
rendering a virtual object based on the pose of the physical object relative to the first reference coordinate system; and
providing the rendered virtual object to the augmented reality display device, wherein the augmented reality display device is configured to display the rendered virtual object based on the pose of the physical object relative to the first reference coordinate system.
17. The server system of claim 11 , wherein the first sensor includes a visual odometry system configured to identify a position and an orientation of the augmented reality display device relative to the first reference coordinate system.
18. The server system of claim 11 , wherein the second sensor includes an optical sensor configured to generate second sensor data based on an image of the physical object captured by the optical sensor.
19. The server system of claim 11 , wherein the operations further comprise:
receiving a request from the augmented reality display device to offload a tracking process of the augmented reality display device to the server system;
in response to receiving the request, generating offloaded processed data based on the first sensor data and second sensor data; and
providing the offloaded processed data to the augmented reality display device, the augmented reality display device configured to present, in a display of the augmented reality display device, virtual objects based on the offloaded processed data.
20. A non-transitory computer-readable medium storing instructions that, when executed by one or more computer processors of a server system, cause the server system to perform operations comprising:
receiving first sensor data from a first sensor of an augmented reality display device, the first sensor data indicating a pose of the augmented reality display device relative to a first reference coordinate system;
receiving second sensor data from a second sensor of the augmented reality display device;
detecting a physical object using the second sensor data;
determining, based on the second sensor data, a pose of the physical object relative to the augmented reality display device;
determining the pose of the physical object relative to the first reference coordinate system based on the pose of the physical object relative to the augmented reality display device and the pose of the augmented reality display device relative to the first reference coordinate system; and
providing the pose of the physical object relative to the first reference coordinate system to the augmented reality display device.