Systems and methods for virtual and augmented reality
Disclosed herein are systems and methods for distributed computing and/or networking for mixed reality systems. A method may include capturing an image via a camera of a head-wearable device. Inertial data may be captured via an inertial measurement unit of the head-wearable device. A position of the head-wearable device can be estimated based on the image and the inertial data via one or more processors of the head-wearable device. The image can be transmitted to a remote server. A neural network can be trained based on the image via the remote server. A trained neural network can be transmitted to the head-wearable device.
1. A method comprising:
capturing, via a sensor of a head-wearable device, local pose data, the local pose data describing an orientation of the head-wearable device;
capturing, via a camera of the head-wearable device, image data;
transmitting the local pose data and the image data to a remote device that is remote to the head-wearable device, such that the remote device, in response to receiving the local pose data and the image data, determines remote pose data based on the local pose and further based on the image data, the remote pose data describing the orientation of the head-wearable device;
receiving, at the head-wearable device from the remote device, the remote pose data; and
determining, at the head-wearable device, a head pose based on the local pose data and based further on the remote pose data.
2. The method of claim 1 , further comprising:
determining, based on the head pose, whether to present a virtual object to a user of the head-wearable device;
in accordance with a determination that the virtual object should be presented, presenting the virtual object to the user via a transmissive display of the head-wearable device; and
in accordance with a determination that the virtual object should not be presented, forgoing presenting the virtual object to the user via the transmissive display.
3. The method of claim 1 , wherein the remote pose data is determined based on an application of a neural network to the local pose data.
4. The method of claim 1 , wherein the remote pose data is determined based on a Simultaneous Localization and Mapping (SLAM) optimization.
5. The method of claim 1 , wherein:
transmitting the local pose data to the remote device comprises transmitting the local pose data via an intermediary relay;
receiving the remote pose data comprises receiving the remote pose data via the intermediary relay;
the intermediary relay comprises a mesh of two or more devices; and
the mesh comprises a smartphone.
6. The method of claim 1 , wherein:
transmitting the local pose data to the remote device comprises transmitting the local pose data via an intermediary relay;
receiving the remote pose data comprises receiving the remote pose data via the intermediary relay;
the intermediary relay comprises a mesh of two or more devices; and
the mesh comprises a drone device.
7. A system comprising:
a head-wearable device comprising a sensor and further comprising a camera; and
one or more processors configured to perform a method comprising:
capturing, via the sensor, local pose data, the local pose data describing an orientation of the head-wearable device;
capturing, via the camera, image data;
transmitting the local pose data and the image data to a remote device that is remote to the head-wearable device, such that the remote device, in response to receiving the local pose data and the image data, determines remote pose data based on the local pose and further based on the image data, the remote pose data describing the orientation of the head-wearable device;
receiving, at the head-wearable device from the remote device, the remote pose data; and
determining, at the head-wearable device, a head pose based on the local pose data and based further on the remote pose data.
8. The system of claim 7 , wherein the method further comprises:
determining, based on the head pose, whether to present a virtual object to a user of the head-wearable device;
in accordance with a determination that the virtual object should be presented, presenting the virtual object to the user via a transmissive display of the head-wearable device; and
in accordance with a determination that the virtual object should not be presented, forgoing presenting the virtual object to the user via the transmissive display.
9. The system of claim 7 , wherein the remote pose data is determined based on an application of a neural network to the local pose data.
10. The system of claim 7 , wherein the remote pose data is determined based on a Simultaneous Localization and Mapping (SLAM) optimization.
11. The system of claim 7 , wherein:
transmitting the local pose data to the remote device comprises transmitting the local pose data via an intermediary relay;
receiving the remote pose data comprises receiving the remote pose data via the intermediary relay;
the intermediary relay comprises a mesh of two or more devices; and
the mesh comprises a smartphone.
12. The system of claim 7 , wherein:
transmitting the local pose data to the remote device comprises transmitting the local pose data via an intermediary relay;
receiving the remote pose data comprises receiving the remote pose data via the intermediary relay;
the intermediary relay comprises a mesh of two or more devices; and
the mesh comprises a drone device.
13. A non-transitory computer-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform a method comprising:
capturing, via a sensor of a head-wearable device, local pose data, the local pose data describing an orientation of the head-wearable device;
capturing, via a camera of the head-wearable device, image data;
transmitting the local pose data and the image data to a remote device that is remote to the head-wearable device, such that the remote device, in response to receiving the local pose data and the image data, determines remote pose data based on the local pose and further based on the image data, the remote pose data describing the orientation of the head-wearable device;
receiving, at the head-wearable device from the remote device, the remote pose data; and
determining, at the head-wearable device, a head pose based on the local pose data and based further on the remote pose data.
14. The non-transitory computer-readable medium of claim 13 , the method further comprising:
determining, based on the head pose, whether to present a virtual object to a user of the head-wearable device;
in accordance with a determination that the virtual object should be presented, presenting the virtual object to the user via a transmissive display of the head-wearable device; and
in accordance with a determination that the virtual object should not be presented, forgoing presenting the virtual object to the user via the transmissive display.
15. The non-transitory computer-readable medium of claim 13 , wherein the remote pose data is determined based on an application of a neural network to the local pose data.
16. The non-transitory computer-readable medium of claim 13 , wherein the remote pose data is determined based on a Simultaneous Localization and Mapping (SLAM) optimization.
17. The non-transitory computer-readable medium of claim 13 , wherein:
transmitting the local pose data to the remote device comprises transmitting the local pose data via an intermediary relay;
receiving the remote pose data comprises receiving the remote pose data via the intermediary relay;
the intermediary relay comprises a mesh of two or more devices; and
the mesh comprises one or more of a smartphone and a drone device.