Systems and methods for virtual and augmented reality
Disclosed herein are systems and methods for distributed computing and/or networking for mixed reality systems. A method may include capturing an image via a camera of a head-wearable device. Inertial data may be captured via an inertial measurement unit of the head-wearable device. A position of the head-wearable device can be estimated based on the image and the inertial data via one or more processors of the head-wearable device. The image can be transmitted to a remote server. A neural network can be trained based on the image via the remote server. A trained neural network can be transmitted to the head-wearable device.
1. A method comprising:
receiving, at a first time, local target data captured via one or more sensors of a first wearable system comprising a first head-wearable device, the local target data comprising a direction associated with the first wearable system;
receiving image data captured via a camera of the first wearable system, the image data associated with a field of view of the first head-wearable device;
in response to receiving the local target data and further in response to receiving the image data, determining trained target data based on the local target data and further based on the image data, wherein said determining the trained target data comprises applying the local target data and the image data as inputs to a convolutional neural network; and
transmitting, to a second wearable system, the trained target data;
wherein the second wearable system is configured to:
detect an object in a field of view of the second wearable system; and
determine, at the second wearable system based on the trained target data, whether the detected object comprises a target object, wherein said determining whether the detected object comprises a target object is based on the convolutional neural network.
2. The method of claim 1 , wherein the one or more sensors comprise a laser and wherein the direction is based on a direction of the laser.
3. The method of claim 1 , wherein the direction is based on an aim direction of a weapon and wherein the one or more sensors comprise a sensor associated with the weapon.
4. The method of claim 1 , further comprising:
receiving, at a second time later than the first time, second local target data;
receiving, at a third time later than the first time, second image data;
in response to receiving the second local target data and further in response to receiving the second image data, determining updated trained target data based on the trained target data and based further on the second local target data and the second image data; and
transmitting, to the second wearable system, the updated trained target data.
5. The method of claim 1 , wherein the second wearable system comprises the first wearable system.
6. The method of claim 1 , wherein the second wearable system comprises a second head-wearable device different from the first head-wearable device.
7. The method of claim 6 , wherein:
the local target data is received at a computing device;
the computing device is located less than 50 miles from the first head-wearable device; and
the computing device is located less than 50 miles from the second head-wearable device.
8. A system comprising:
a computing device comprising one or more processors, the one or more processors configured to perform a method comprising:
receiving, at a first time, local target data captured via one or more sensors of a first wearable system comprising a first head-wearable device, the local target data comprising a direction associated with the first wearable system;
receiving image data captured via a camera of the first wearable system, the image data associated with a field of view of the first head-wearable device;
in response to receiving the local target data and further in response to receiving the image data, determining trained target data based on the local target data and further based on the image data, wherein said determining the trained target data comprises applying the local target data and the image data as inputs to a convolutional neural network; and
transmitting, to a second wearable system, the trained target data;
wherein the second wearable system is configured to:
detect an object in a field of view of the second wearable system; and
determine, at the second wearable system based on the trained target data, whether the detected object comprises a target object, wherein said determining whether the detected object comprises a target object is based on the convolutional neural network.
9. The system of claim 8 , wherein the first wearable system comprises the computing device.
10. The system of claim 8 , wherein the direction is based on an aim direction of a weapon and wherein the one or more sensors comprise a sensor associated with the weapon.
11. The system of claim 8 , wherein the method further comprises:
receiving, at a second time later than the first time, second local target data;
receiving, at a third time later than the first time, second image data;
in response to receiving the second local target data and further in response to receiving the second image data, determining updated trained target data based on the trained target data and based further on the second local target data and the second image data; and
transmitting, to the second wearable system, the updated trained target data.
12. The system of claim 8 , wherein the second wearable system comprises the first wearable system.
13. The system of claim 8 , wherein the second wearable system comprises a second head-wearable device different from the first head-wearable device.
14. The system of claim 13 , wherein:
the computing device is located less than 50 miles from the first head-wearable device; and
the computing device is located less than 50 miles from the second head-wearable device.
15. A non-transitory computer-readable storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform a method comprising:
receiving, at a first time, local target data captured via one or more sensors of a first wearable system comprising a first head-wearable device, the local target data comprising a direction associated with the first wearable system;
receiving image data captured via a camera of the first wearable system, the image data associated with a field of view of the first head-wearable device;
in response to receiving the local target data and further in response to receiving the image data, determining trained target data based on the local target data and further based on the image data, wherein said determining the trained target data comprises applying the local target data and the image data as inputs to a convolutional neural network; and
transmitting, to a second wearable system, the trained target data;
wherein the second wearable system is configured to:
detect an object in a field of view of the second wearable system; and
determine, at the second wearable system based on the trained target data, whether the detected object comprises a target object, wherein said determining whether the detected object comprises a target object is based on the convolutional neural network.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the direction is based on an aim direction of a weapon and wherein the one or more sensors comprise a sensor associated with the weapon.
17. The non-transitory computer-readable storage medium of claim 15 , wherein the method further comprises:
receiving, at a second time later than the first time, second local target data;
receiving, at a third time later than the first time, second image data;
in response to receiving the second local target data and further in response to receiving the second image data, determining updated trained target data based on the trained target data and based further on the second local target data and the second image data; and
transmitting, to the second wearable system, the updated trained target data.
18. The non-transitory computer-readable storage medium of claim 15 , wherein the second wearable system comprises the first wearable system.
19. The non-transitory computer-readable storage medium of claim 15 , wherein the second wearable system comprises a second head-wearable device different from the first head-wearable device.
20. The non-transitory computer-readable storage medium of claim 15 , wherein:
the local target data is received at a computing device;
the computing device is located less than 50 miles from the first head-wearable device; and
the computing device is located less than 50 miles from the second head-wearable device.