Lightweight and low power cross reality device with high temporal resolution
A wearable display system for a cross reality (XR) system may have a dynamic vision sensor (DVS) camera and a color camera. At least one of the cameras may be a plenoptic camera. The wearable display system may dynamically restrict processing of image data from either or both cameras based on detected conditions and XR function being performed. For tracking an object, image information may be processed for patches of a field of view of either or both cameras corresponding to the object. The object may be tracked based on asynchronously acquired events indicating changes within the patches. Stereoscopic or other types of image information may be used when event-based object tacking yields an inadequate quality metric. The tracked object may be a user's hand or a stationary object in the physical world, enabling calculation of the pose of the wearable display system and of the wearer's head.
1. A wearable display system, the wearable display system comprising:
a headset including:
a first camera configurable to output an image frame or image data satisfying an intensity change criterion; and
a second camera,
wherein the first camera and the second camera are positioned so as to provide overlapping views of a central view field; and
a processor operatively coupled to the first camera and the second camera and configured to:
create a world model using depth information stereoscopically determined from image data output by the first camera and image data output by the second camera;
perform a tracking routine using the world model and the image data output by the first camera;
determine a tracking parameter for the tracking routine using image data output by at least one of the first camera or the second camera; and
based on the tracking parameter, provide instructions to at least one of the first camera or the second camera to adjust image data acquisition, the instructions configured to increase or decrease an amount of data output by at least one of the first camera or the second camera.
2. The wearable display system of claim 1 , wherein the intensity change criterion comprises an absolute or relative intensity change criterion.
3. The wearable display system of claim 1 , wherein the first camera is configured to output the image data asynchronously.
4. The wearable display system of claim 3 , wherein the processor is further configured to perform the tracking routine asynchronously.
5. The wearable display system of claim 1 , wherein:
providing the instructions to the at least one of the first camera or the second camera to adjust image data acquisition includes providing instructions to at least one of the first camera or the second camera to restrict image data acquisition to points of interest within the world model.
6. The wearable display system of claim 5 , wherein:
the first camera is configurable to restrict image data acquisition to one or more portions of a field of view of the first camera;
determining the tracking parameter includes:
identifying a point of interest within the world model; and
determining one or more first portions of the field of view of the first camera corresponding to the point of interest; and
providing instructions to the at least one of the first camera or the second camera to adjust image data acquisition includes providing instructions to the first camera to restrict image data acquisition to the one or more first portions of the field of view.
7. The wearable display system of claim 6 , wherein:
determining the tracking parameter further includes estimating one or more second portions of the field of view of the first camera corresponding to the point of interest based on a movement of the point of interest relative to the world model or a movement of the headset relative to the point of interest; and
providing the instructions to the at least one of the first camera or the second camera to adjust image data acquisition further includes providing instructions to the first camera to restrict image data acquisition to the one or more second portions of the field of view.
8. The wearable display system of claim 5 , wherein:
the headset further includes an inertial measurement unit; and
determining the tracking parameter further includes estimating an updated relative position of a point of interest of the points of interest based at least partly upon an output of the inertial measurement unit.
9. The wearable display system of claim 5 , wherein:
determining the tracking parameter further includes repeatedly calculating a position of the point of interest within the world model; and
repeated calculations are performed at a temporal resolution of more than 60 Hz.
10. The wearable display system of claim 9 , wherein intervals between the repeated calculations are between 1 ms and 15 ms in duration.
11. The wearable display system of claim 1 , wherein:
determining the tracking parameter includes determining whether tracking satisfies a quality criterion; and
providing the instructions to the at least one of the first camera or the second camera to adjust image data acquisition includes providing instructions to the second camera to enable the second camera or modulate a frame rate of the second camera when the tracking does not satisfy the quality criterion.
12. The wearable display system of claim 1 , wherein the processor is mechanically coupled to the headset.
13. The wearable display system of claim 1 , wherein the headset comprises a display device mechanically coupled to the processor.
14. The wearable display system of claim 1 , wherein a local data processing module comprises the processor, the local data processing module operatively coupled to a display device through a communication link, and wherein the headset comprises the display device.
15. The wearable display system of claim 1 , wherein the headset further includes an IR emitter.
16. The wearable display system of claim 15 , wherein the processor is configured to selectively enable the IR emitter so as to enable tracking in a low light condition.
17. A method of cross reality tracking using a wearable display system, the wearable display system comprising:
a headset including:
a first camera configurable to output an image frame or image data satisfying an intensity change criterion; and
a second camera,
wherein the first camera and the second camera are positioned so as to provide overlapping views of a central view field; and
a processor operatively coupled to the first camera and the second camera,
the method comprising:
using the processor to:
create a world model using depth information stereoscopically determined from image data output by the first camera and image data output by the second camera;
performing a tracking routine using the world model and the image data output by the first camera;
determine a tracking parameter for the tracking routine using image data output by at least one of the first camera or the second camera; and
based on the tracking parameter, provide instructions to at least one of the first camera or the second camera to adjust image data acquisition, the instructions configured to increase or decrease an amount of data output by at least one of the first camera or the second camera.
18. The method of claim 17 , wherein:
providing the instructions to the at least one of the first camera or the second camera to adjust image data acquisition includes providing instructions to at least one of the first camera or the second camera to restrict image data acquisition to points of interest within the world model.
19. The method of claim 18 , wherein:
determining the tracking parameter includes:
identifying a point of interest within the world model; and
determining one or more first portions of a field of view of the first camera corresponding to the point of interest; and
providing instructions to the first camera to restrict image data acquisition to the one or more first portions of the field of view.
20. The method of claim 19 , wherein:
determining the tracking parameter further includes estimating one or more second portions of the field of view of the first camera corresponding to the point of interest based on a movement of the point of interest relative to the world model or a movement of the headset relative to the point of interest; and
providing the instructions to the at least one of the first camera or the second camera to adjust image data acquisition further includes providing instructions to the first camera to restrict image data acquisition to the one or more second portions of the field of view.