Minimal-latency tracking and display for matching real and virtual worlds in head-worn displays
Methods, systems, and computer readable media for minimal-latency tracking and display for matching real and virtual worlds in head-worn displays are disclosed. According to one aspect, a method for minimal-latency tracking and display for matching real and virtual worlds in head-worn displays includes calculating a desired image, calculating an error image as the difference between the desired image and an image currently being perceived by a user, identifying as an error portion a portion of the error image having the largest error, updating a portion of a projected image that corresponds to the error portion, and recalculating the image currently being perceived by a user based on the updated projected image.
1. A system for minimal-latency tracking and display for matching real and virtual worlds in head-worn displays, the system comprising:
a display;
at least one processor;
a display control module for controlling projection of an image by the display to a user;
a tracking module for obtaining pose information regarding the user; and
a rendering and processing module for calculating a desired image based on the pose information determined by the tracking module and storing image data regarding the desired image, calculating an estimated user-perceived image, wherein calculating the estimated user-perceived image includes:
storing image data regarding scanline updates previously displayed by the display to the user;
storing image data regarding the current image projected to the user by the display; and
calculating, using the image data regarding scanline updates previously projected to the user and the image data regarding the current image projected by the display to the user, image data estimating an image that the user is currently perceiving,
wherein the rendering and processing module identifies, by calculating an error comprising a difference in at least one of perceived color and intensity between data corresponding to rows or other portions of the estimated user-perceived image in the image data estimating the image that the user is currently perceiving and data corresponding to rows or other portions of the desired image in the image data regarding the desired image, a row or other portion of the image currently being projected by the display having a largest error; and
wherein the display control module updates the row or other portion of the image currently being projected by the display identified as having the largest error.
2. The system of claim 1 wherein the rendering and processing module recalculates the estimated user-perceived image based on an updated projection image projected by the display and a model of human perception.
3. The system of claim 1 wherein the display comprises a head-wearable display, wherein the display control module is a component of the head-wearable display.
4. The system of claim 3 wherein the head-wearable display comprises an optical see-through display.
5. The system of claim 3 wherein the head-wearable display comprises a digital micromirror device (DMD).
6. The system of claim 5 wherein the head-wearable display module includes an illumination source for illuminating the DMD.
7. The system of claim 6 wherein the illumination source includes multiple color filters for modifying a hue of illumination of pixels that form the scanlines.
8. The system of claim 7 wherein updating the scanlines includes changing the hue and/or intensity of the illumination of the pixels that form the scan lines.
9. The system of claim 6 wherein the illumination source includes a multicolor illumination source.
10. The system of claim 9 wherein the multicolor illumination source comprises a light emitting diode (LED).
11. The system of claim 1 wherein the tracking module calculates pose based on row data received from a rolling shutter camera.
12. A method for minimal-latency tracking and display for matching real and virtual worlds in head-worn displays, the method comprising:
projecting, by a display, an image to a user;
calculating an estimated user-perceived image, wherein calculating the estimated user-perceived image includes:
storing image data regarding scanline updates previously projected by the display to the user;
storing image data regarding the image currently being projected by the display to the user; and
calculating, using the image data regarding scanline updates previously projected by the display to the user and the image data regarding the image currently being projected by the display to the user, image data estimating an image that the user is currently perceiving;
calculating a desired image based on pose information regarding the user and storing image data regarding the desired image;
identifying, by calculating an error comprising a difference in at least one of perceived color and intensity between data corresponding to rows or other portions of the estimated user-perceived image in the image data estimating the image that the user is currently perceiving and data corresponding to rows or other portions of the desired image in the image data regarding the desired image, a row or other portion of the image currently being projected by the display having a largest error; and
updating the row or other portion of the image currently being projected by the display identified as having the largest error.
13. The method of claim 12 comprising recalculating the estimated user-perceived image based on an updated image projected by the display and a model of human perception.
14. The method of claim 12 wherein the display comprises a head-mounted display.
15. The method of claim 12 wherein the display comprises an optical see-through display.
16. The method of claim 12 wherein calculating a desired image comprises receiving the image data regarding the desired image from a rendering module.
17. The method of claim 12 comprising determining importance values for features of the desired image and using the importance values to prioritize updating of the scanlines corresponding to the features.
18. The method of claim 12 comprising calculating the error image by calculating differences in hue and/or intensity between the image data regarding the desired image and the image data estimating the image that the user is currently perceiving.
19. The method of claim 12 wherein the pose information comprises row data received from a rolling shutter camera.
20. A non-transitory computer readable medium having stored thereon executable instructions that when executed by the processor of a computer control the computer to perform steps comprising:
projecting, by display, an image to a user;
calculating an estimated user-perceived image, wherein calculating the estimated user-perceived image includes:
storing image data regarding scanline updates previously displayed by a display to the user;
storing image data regarding the image currently being projected to the user by the display; and
calculating, using the image data regarding scanline updates previously projected by the display to the user and the image data regarding the image currently being projected by the display to the user, image data estimating an image that the user is currently perceiving;
calculating a desired image based on pose information regarding the user and storing image data regarding the desired image;
identifying, by calculating an error comprising a difference in at least one of perceived color and intensity between data corresponding to rows or other portions in the image data estimating the image that the user is currently perceiving and data corresponding to rows or other portions of the desired image in the image data regarding the desired image, a row or other portion of the image currently being projected by the display having a largest error; and
updating the row or other portion of the image currently being projected by the display identified as having the largest error.