Head-mounted display for virtual and mixed reality with inside-out positional, user body and environment tracking
A Head-Mounted Display system together with associated techniques for performing accurate and automatic inside-out positional, user body and environment tracking for virtual or mixed reality are disclosed. The system uses computer vision methods and data fusion from multiple sensors to achieve real-time tracking. High frame rate and low latency is achieved by performing part of the processing on the HMD itself.
1. A method comprising:
at a head-mounted device (HMD) including non-transitory memory, one or more processors, and a communications interface for communicating with first and second RGB stereo-cameras, an infrared (IR) projector, one or more IR sensors, and a display;
obtaining, via the first RGB stereo-camera, a first stereo image of a physical environment;
obtaining, via the second RGB stereo-camera, a second stereo image of the physical environment;
projecting, via the IR projector, an IR pattern of IR electromagnetic radiation onto the physical environment;
obtaining, via the one or more IR sensors, texture information for the physical environment associated with a reflection of the IR pattern;
generating a dense depth map by matching each pixel in the first stereo image to a different pixel in the second stereo image based at least in part on the texture information;
generating a display image by compositing virtual content with the first and second stereo image based on the dense depth map; and
displaying, via the display, the display image.
2. The method of claim 1 , wherein the first and second RGB stereo-cameras and the one or more IR camera sensors share a common axis.
3. The method of claim 1 , further comprising:
generating, via the IR projector, the IR pattern according to one of a random or pseudo-random algorithm.
4. The method of claim 1 , wherein a granularity of the IR pattern is adjusted by one of: (A) focusing the IR pattern on a spot of a different size on a diffusing surface or (B) changing the diffusing surface.
5. The method of claim 1 , wherein the IR pattern corresponds to an interference pattern of a laser beam passing through a surface diffuser.
6. The method of claim 1 , wherein the IR pattern corresponds to a far field diffraction of a laser beam passing through one or many diffractive optical elements.
7. The method of claim 1 , wherein the IR speckle projector includes a polarizer.
8. The method of claim 1 , wherein the communications interface of the HMD also communicated with one or more IR flood lights, and the method further comprising:
operating the IR projector and the one or more IR flood lights according to a temporal multiplexing scheme.
9. The method of claim 8 , further comprising:
modulating at least one of exposure time, sensor gain, and intensity of the IR projector and the one or more IR flood lights in real-time to reduce image saturation and improve dynamic range.
10. The method of claim 1 , wherein the first RGB stereo-camera includes a first IR sensor among the one or more IR sensor, and wherein the first IR sensor corresponds to a near IR bandpass filter.
11. A head-mounted device (HMD) comprising:
a communications interface for communicating with first and second RGB stereo-cameras, an infrared (IR) projector, one or more IR sensors, and a display;
one or more processors; and
a non-transitory memory storing one or more programs, which, when executed by the one or more processors, cause the HMD to:
obtain, via the first RGB stereo-camera, a first stereo image of a physical environment;
obtain, via the second RGB stereo-camera, a second stereo image of the physical environment;
project, via the IR projector, an IR pattern of IR electromagnetic radiation onto the physical environment;
obtain, via the one or more IR sensors, texture information for the physical environment associated with a reflection of the IR pattern;
generate a dense depth map by matching each pixel in the first stereo image to a different pixel in the second stereo image based at least in part on the texture information;
generate a display image by compositing virtual content with the first and second stereo image based on the dense depth map; and
display, via the display, the display image.
12. The HMD of claim 11 , wherein the first and second RGB stereo-cameras and the one or more IR camera sensors share a common axis.
13. The HMD of claim 11 , wherein the first RGB stereo-camera includes a first IR sensor among the one or more IR sensor, and wherein the first IR sensor corresponds to a near IR bandpass filter.
14. The HMD of claim 11 , wherein the communications interface of the HMD also communicated with one or more IR flood lights, and
wherein the one or more programs further cause the HMD to:
operate the IR projector and the one or more IR flood lights according to a temporal multiplexing scheme.
15. The HMD of claim 14 , wherein the one or more programs further cause the HMD to:
modulate at least one of exposure time, sensor gain, and intensity of the IR projector and the one or more IR flood lights in real-time to reduce image saturation and improve dynamic range.
16. A non-transitory computer-readable medium having instructions encoded thereon, which, when executed by one or more processors of a head-mounted device (HMD) including a communications interface for communicating with first and second RGB stereo-cameras, an infrared (IR) projector, one or more IR sensors, and a display, cause the HMD to:
obtain, via the first RGB stereo-camera, a first stereo image of a physical environment;
obtain, via the second RGB stereo-camera, a second stereo image of the physical environment;
project, via the IR projector, an IR pattern of IR electromagnetic radiation onto the physical environment;
obtain, via the one or more IR sensors, texture information for the physical environment associated with a reflection of the IR pattern;
generate a dense depth map by matching each pixel in the first stereo image to a different pixel in the second stereo image based at least in part on the texture information;
generate a display image by compositing virtual content with the first and second stereo image based on the dense depth map; and
display, via the display, the display image.
17. The non-transitory computer-readable medium of claim 16 , wherein the first and second RGB stereo-cameras and the one or more IR camera sensors share a common axis.
18. The non-transitory computer-readable medium of claim 16 , wherein the first RGB stereo-camera includes a first IR sensor among the one or more IR sensor, and wherein the first IR sensor corresponds to a near IR bandpass filter.
19. The non-transitory computer-readable medium of claim 16 , wherein the communications interface of the HMD also communicated with one or more IR flood lights, and
wherein the instructions further cause the HMD to:
operate the IR projector and the one or more IR flood lights according to a temporal multiplexing scheme.
20. The non-transitory computer-readable medium of claim 19 , wherein the instructions further cause the HMD to:
modulate at least one of exposure time, sensor gain, and intensity of the IR projector and the one or more IR flood lights in real-time to reduce image saturation and improve dynamic range.