Shared data and collaboration for head-mounted devices
A system can include head-mounted devices that collaborate to process views from cameras of the respective head-mounted devices and identify objects from different perspectives and/or objects that are within the view of only one of the head-mounted devices. Sharing sensory input between multiple head-mounted devices can complement and enhance individual units by interpreting and reconstructing objects, surfaces, and/or an external environment with perceptive data from multiple angles and positions, which also reduces occlusions and inaccuracies. As more detailed information is available at a specific moment in time, the speed and accuracy of object recognition, hand and body tracking, surface mapping, and/or digital reconstruction can be improved. Such collaboration can provide more effective and efficient mapping of space, surfaces, objects, gestures and users.
1. A head-mounted device comprising:
a camera for capturing a first view including a first physical object;
a display;
a communication element for receiving data from an additional head-mounted device comprising an additional camera for capturing a second view including a second physical object that is not in the first view, wherein the additional head-mounted device is within the first view; and
a processor configured to:
receive, from the additional head-mounted device, the data, the data being based on the second view; and
determine a characteristic of the first physical object based on the first view captured by the camera and the data from the additional head-mounted device;
transmit, to the additional head-mounted device, a signal indicating the characteristic of the first physical object;
operate the display to output a virtual object as a substitute for the first physical object based on the characteristic of the first physical object;
output a representation of the second physical object based on the data received from the additional head-mounted device; and
determine whether, based on the first view, the additional camera is arranged to capture the second view of the first physical object.
2. The head-mounted device of claim 1 , wherein the characteristic comprises an identity of the first physical object.
3. The head-mounted device of claim 1 , wherein the processor is further configured to:
determine whether the additional camera of the additional head-mounted device is arranged to capture the second view of the first physical object; and
transmit, with the communication element and to the additional head-mounted device, a request for the data based on the second view captured by the additional camera.
4. The head-mounted device of claim 1 , wherein the processor is further configured to:
receive, with the communication element, arrangement information of the additional head-mounted device; and
determine whether, based on the arrangement information, the additional camera is arranged to capture the second view of the first physical object is.
5. A system comprising:
the head-mounted device of claim 1 ;
the additional head-mounted device; and
an external device in communication with the head-mounted device and the additional head-mounted device.
6. The system of claim 5 , wherein the data is transmitted from the additional head-mounted device to the head-mounted device via the external device, and the signal is transmitted from the head-mounted device to the additional head-mounted device via the external device.
7. The system of claim 5 , wherein the external device is configured to:
determine an additional characteristic of the first physical object based on the first view and the data; and
transmit, to the head-mounted device and the additional head-mounted device, an additional signal indicating the additional characteristic of the first physical object.
8. The system of claim 5 , wherein the external device is configured to:
receive the data from the additional head-mounted device;
store the data until the camera captures the first view including the first physical object; and
transmit the data to the head-mounted device.
9. A head-mounted device comprising:
a camera for capturing a first view including a first physical object;
a display; and
a processor configured to:
receive, with a communication element and from an additional head-mounted device, data based on a second view captured by an additional camera of the additional head-mounted device, the second view including the first physical object and a second physical object, the second physical object not being in the first view;
determine a characteristic of the first physical object based on the first view captured by the camera and the data from the additional head-mounted device;
transmit, with the communication element and to the additional head-mounted device, a signal indicating the characteristic of the first physical object;
operate the display to output the first view with a virtual object that replaces the first physical object in the first view, the virtual object being based on the characteristic of the first physical object;
output a representation of the second physical object based on the data received from the additional head-mounted device;
determine whether the additional camera of the additional head-mounted device is arranged to capture the second view of the first physical object; and
transmit, with the communication element and to the additional head-mounted device, a request for the data.
10. The head-mounted device of claim 9 , wherein:
the additional head-mounted device is within the first view; and
the processor is further configured to determine whether, based on the first view, the additional camera is arranged to capture the second view of the first physical object.
11. The head-mounted device of claim 9 , wherein the processor is further configured to:
receive, with the communication element, arrangement information of the additional head-mounted device; and
determine whether, based on the arrangement information, the additional camera is arranged to capture the second view of the first physical object is.
12. The head-mounted device of claim 9 , wherein the characteristic comprises an identity of the first physical object.
13. A system comprising:
the head-mounted device of claim 9 ;
the additional head-mounted device; and
an external device in communication with the head-mounted device and the additional head-mounted device.
14. The system of claim 13 , wherein the data is transmitted from the additional head-mounted device to the head-mounted device via the external device, and the signal is transmitted from the head-mounted device to the additional head-mounted device via the external device.
15. A head-mounted device comprising:
a camera;
a display;
a communication element; and
a processor configured to:
capture a first view of a first physical object with the camera;
receive, with the communication element, data from an additional head-mounted device comprising an additional camera, the data being based on a second view of the first physical object captured by the additional camera, the second view further including a second physical object that is not in the first view;
determine a characteristic of the first physical object based on the first view and the data;
transmit, with the communication element and to the additional head-mounted device, a signal indicating the characteristic of the first physical object;
operate the display to output a virtual object as a substitute for the first physical object based on the characteristic of the first physical object; and
output a representation of the second physical object based on the data received from the additional head-mounted device; and
receive, with the communication element, arrangement information of the additional head-mounted device; and
determine whether, based on the arrangement information, the additional camera is arranged to capture the second view of the first physical object.
16. The head-mounted device of claim 15 , wherein the characteristic comprises an identity of the first physical object.
17. The head-mounted device of claim 15 , wherein the processor is further configured to:
determine whether the additional camera of the additional head-mounted device is arranged to capture the second view of the first physical object; and
transmit, with the communication element and to the additional head-mounted device, a request for the data based on the second view captured by the additional camera.