Systems and methods for virtual and augmented reality
Disclosed herein are systems and methods for distributed computing and/or networking for mixed reality systems. A method may include capturing an image via a camera of a head-wearable device. Inertial data may be captured via an inertial measurement unit of the head-wearable device. A position of the head-wearable device can be estimated based on the image and the inertial data via one or more processors of the head-wearable device. The image can be transmitted to a remote server. A neural network can be trained based on the image via the remote server. A trained neural network can be transmitted to the head-wearable device.
1. A method comprising:
capturing, via one or more sensors of a wearable system comprising a head-wearable device, local target data, the local target data comprising a direction associated with the wearable system;
capturing, via a camera of the head-wearable device, image data associated with a field of view of the head-wearable device;
transmitting the local target data and the image data to a remote device that is remote to the head-wearable device, wherein the remote device is configured to, in response to receiving the local target data and the image data, determine trained target data based on the local target data and further based on the image data, wherein said determining the trained target data comprises applying the local target data and the image data as inputs to a convolutional neural network;
receiving, at the head-wearable device from the remote device, the trained target data;
detecting, via the one or more sensors, an object in the field of view; and
determining, at the head-wearable device based on the trained target data, whether the detected object comprises a target object, wherein said determining whether the detected object comprises a target object is based on an output of the convolutional neural network.
2. The method of claim 1 , wherein the one or more sensors comprise a laser and wherein the direction associated with the wearable system comprises a direction of the laser.
3. The method of claim 1 , wherein the direction associated with the wearable system comprises an aim direction of a weapon and wherein the one or more sensors comprise a sensor associated with the weapon.
4. The method of claim 1 , further comprising receiving, at the head-wearable device from the remote device, updated trained target data,
wherein the remote device is configured to determine the updated trained target data based on the trained target data and based further on second local target data and second image data.
5. The method of claim 1 , further comprising:
receiving, at a second head-wearable device from the remote device, the trained target data;
detecting, via one or more sensors of the second head-wearable device, a second object in the field of view of a user of the second head-wearable device; and
determining, at the second head-wearable device based on the trained target data, whether the detected second object comprises a target object.
6. The method of claim 5 , wherein:
the head-wearable device is in communication with the second head-wearable device;
the remote device is located less than 50 miles from the head-wearable device; and
the remote device is located less than 50 miles from the second head-wearable device.
7. A system comprising:
one or more sensors;
a head-wearable device comprising a camera; and
one or more processors configured to perform a method comprising:
capturing, via the one or more sensors, local target data, the local target data comprising a direction associated with the system;
capturing, via the camera, image data associated with a field of view of the head-wearable device;
transmitting the local target data and the image data to a remote device that is remote to the head-wearable device, wherein the remote device is configured to, in response to receiving the local target data and the image data, determine trained target data based on the local target data and further based on the image data, wherein said determining the trained target data comprises applying the local target data and the image data as inputs to a convolutional neural network;
receiving, at the head-wearable device from the remote device, the trained target data;
detecting, via the one or more sensors, an object in the field of view; and
determining, at the head-wearable device based on the trained target data, whether the detected object comprises a target object, wherein said determining whether the detected object comprises a target object is based on an output of the convolutional neural network.
8. The system of claim 7 , wherein the one or more sensors comprise a laser and wherein the direction associated with the system comprises a direction of the laser.
9. The system of claim 7 , wherein the direction associated with the system comprises an aim direction of a weapon and wherein the one or more sensors comprise a sensor associated with the weapon.
10. The system of claim 7 , wherein:
the method further comprises receiving, at the head-wearable device from the remote device, updated trained target data, and
the remote device is configured to determine the updated trained target data based on the trained target data and based further on second local target data and second image data.
11. The system of claim 7 , further comprising a second head-wearable device, wherein the method further comprises:
receiving, at the second head-wearable device from the remote device, the trained target data;
detecting, via one or more sensors of the second head-wearable device, a second object in the field of view of a user of the second head-wearable device; and
determining, at the second head-wearable device based on the trained target data, whether the detected second object comprises a target object.
12. A non-transitory computer-readable storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform a method comprising:
capturing, via one or more sensors of a wearable system comprising a head-wearable device, local target data, the local target data comprising a direction associated with the wearable system;
capturing, via a camera of the head-wearable device, image data associated with a field of view of the head-wearable device;
transmitting the local target data and the image data to a remote device that is remote to the head-wearable device, wherein the remote device is configured to, in response to receiving the local target data and the image data, determine trained target data based on the local target data and further based on the image data, wherein said determining the trained target data comprises applying the local target data and the image data as inputs to a convolutional neural network;
receiving, at the head-wearable device from the remote device, the trained target data;
detecting, via the one or more sensors, an object in the field of view; and
determining, at the head-wearable device based on the trained target data, whether the detected object comprises a target object, wherein said determining whether the detected object comprises a target object is based on an output of the convolutional neural network.
13. The non-transitory computer-readable storage medium of claim 12 , wherein the one or more sensors comprise a laser and wherein the direction associated with the wearable system comprises a direction of the laser.
14. The non-transitory computer-readable storage medium of claim 12 , wherein the direction associated with the wearable system comprises an aim direction of a weapon and wherein the one or more sensors comprise a sensor associated with the weapon.
15. The non-transitory computer-readable storage medium of claim 12 , wherein:
the method further comprises receiving, at the head-wearable device from the remote device, updated trained target data, and
the remote device is configured to determine the updated trained target data based on the trained target data and based further on second local target data and second image data.
16. The non-transitory computer-readable storage medium of claim 12 , wherein the method further comprises:
receiving, at a second head-wearable device from the remote device, the trained target data;
detecting, via one or more sensors of the second head-wearable device, a second object in the field of view of a user of the second head-wearable device; and
determining, at the second head-wearable device based on the trained target data, whether the detected second object comprises a target object.