Shared data and collaboration for head-mounted devices
A system can include head-mounted devices that collaborate to process views from cameras of the respective head-mounted devices and identify objects from different perspectives and/or objects that are within the view of only one of the head-mounted devices. Sharing sensory input between multiple head-mounted devices can complement and enhance individual units by interpreting and reconstructing objects, surfaces, and/or an external environment with perceptive data from multiple angles and positions, which also reduces occlusions and inaccuracies. As more detailed information is available at a specific moment in time, the speed and accuracy of object recognition, hand and body tracking, surface mapping, and/or digital reconstruction can be improved. Such collaboration can provide more effective and efficient mapping of space, surfaces, objects, gestures and users.
1. A head-mounted device comprising:
a first camera for capturing a first view including a first object and an additional head-mounted device that comprises a second camera;
a communication element;
a display; and
a processor configured to:
determine, based at least in part on the first view, whether the second camera of the additional head-mounted device is arranged to capture a second view including the first object, the second view being different than the first view, wherein the second view further includes a second object that is not in the first view;
transmit, with the communication element and to the additional head-mounted device, a request for data, the data indicating a characteristic of the first object and a characteristic of the second object based on the second view;
receive, with the communication element, the data from the additional head-mounted device; and
output, with the display, a representation of the first object based on the first view and the data received from the additional head-mounted device and a representation of the second object based on the data received from the additional head-mounted device.
2. The head-mounted device of claim 1 , wherein the processor is further configured to receive, with the communication element and from the additional head-mounted device, arrangement information of the additional head-mounted device, wherein whether the second camera of the additional head-mounted device is arranged to capture the second view including the first object is determined further based on the arrangement information.
3. The head-mounted device of claim 1 , wherein the representation of the first object is a virtual object that is a substitute for the first object in the first view.
4. The head-mounted device of claim 1 , further comprising a sensor configured to detect a position and an orientation of the additional head-mounted device with respect to the first object.
5. The head-mounted device of claim 1 , wherein the first view is captured before the second view and the data is stored on an external device.
6. The head-mounted device of claim 5 , wherein:
the request for the data is transmitted via the external device; and
the data is received via the external device.
7. The head-mounted device of claim 1 , wherein the characteristic of the first object comprises an identity of the first object.
8. A head-mounted device comprising:
a first camera for capturing a first view including a first object;
a communication element configured to receive, from an additional head-mounted device that comprises a second camera, arrangement information of the additional head-mounted device;
a display; and
a processor configured to:
determine, based on the arrangement information, whether the second camera of the additional head-mounted device is arranged to capture a second view including the first object, the second view being different than the first view, wherein the second view further includes a second object that is not in the first view;
transmit, with the communication element and to the additional head-mounted device, a request for data, the data indicating a characteristic of the first object and a characteristic of the second object based on the second view;
receive, with the communication element, the data from the additional head-mounted device; and
output, with the display, a representation of the first object based on the first view and the data received from the additional head-mounted device and a representation of the second object based on the data received from the additional head-mounted device.
9. The head-mounted device of claim 8 , wherein:
the first view further includes the additional head-mounted device; and
whether the second camera of the additional head-mounted device is arranged to capture the second view including the first object is determined further based on the first view.
10. The head-mounted device of claim 8 , wherein the representation of the first object is a virtual object that is a substitute for the first object in the first view.
11. The head-mounted device of claim 8 , wherein the first view is captured before the second view and the data is stored on an external device.
12. The head-mounted device of claim 11 , wherein:
the request for the data is transmitted via the external device; and
the data is received via the external device.
13. The head-mounted device of claim 8 , wherein the characteristic of the first object comprises an identity of the first object.
14. A head-mounted device comprising:
a first camera for capturing a first view including a first physical object;
a communication element;
a display; and
a processor configured to:
determine whether a second camera of an additional head-mounted device is arranged to capture a second view including the first physical object, the second view being different than the first view, wherein the second view further includes a second physical object that is not in the first view;
transmit, with the communication element and to the additional head-mounted device, a request for data, the data indicating a characteristic of the first physical object and a characteristic of the second physical object based on the second view;
receive, with the communication element, the data from the additional head-mounted device; and
output, with the display, a virtual object as a substitute for the first physical object based on the first view and the data received from the additional head-mounted device and a representation of the second physical object based on the data received from the additional head-mounted device.
15. The head-mounted device of claim 14 , wherein the processor is further configured to receive, with the communication element and from the additional head-mounted device, arrangement information of the additional head-mounted device, wherein whether the second camera of the additional head-mounted device is arranged to capture the second view including the first physical object is determined further based on the arrangement information.
16. The head-mounted device of claim 14 , wherein:
the first view further includes the additional head-mounted device; and
whether the second camera of the additional head-mounted device is arranged to capture the second view including the first physical object is determined further based on the first view.
17. The head-mounted device of claim 14 , further comprising a sensor configured to detect a position and an orientation of the additional head-mounted device with respect to the first physical object.
18. The head-mounted device of claim 14 , wherein the first view is captured before the second view and the data is stored on an external device.
19. The head-mounted device of claim 18 , wherein:
the request for the data is transmitted via the external device; and
the data is received via the external device.
20. The head-mounted device of claim 14 , wherein the characteristic of the first physical object comprises an identity of the first physical object.