Real-time multifocal displays with gaze-contingent rendering and optimization
Systems and methods for displaying an image across a plurality of displays are described herein. Pixel intensity values in the multifocal display are determined using correlation values and numerical iterations. An eye tracking system measures eye tracking information about a position of a user's eye, and the pixel intensity values are modified based on the eye tracking information. An image is displayed on the plurality of displays based on the determined pixel intensity values. The plurality of displays may be within an HMD, and address vergence accommodation conflict by simulating retinal defocus blur.
1. A method comprising:
determining a first pixel intensity value for each pixel of a plurality of pixels by applying at least one numerical iteration on a plurality of correlation values related to the plurality of pixels;
determining a second pixel intensity value for each pixel of the plurality of pixels using the first intensity value and applying one or more numerical iterations;
determining information about a gaze direction for an eye;
modifying the second pixel intensity value based on the information about the gaze direction; and
determining a pixel intensity value for each pixel of the plurality of pixels to generate content for presentation, based on the modified second pixel intensity value.
2. The method of claim 1 , wherein the pixels belong to at least two displays located at a different focal distance from one another.
3. The method of claim 1 , further comprising:
obtaining a blur gradient correlation value based on the correlation values; and
determining the first pixel intensity value by applying the at least one numerical iteration on the blur gradient correlation value.
4. The method of claim 3 , wherein obtaining the blur gradient correlation value comprises:
subtracting first correlation values of the plurality of correlation values in a first kernel corresponding to a first focal distance from second correlation values of the plurality of correlation values in a second kernel corresponding to a second focal distance; and
determining the blur gradient correlation value based on the subtraction.
5. The method of claim 1 , further comprising:
determining the first pixel intensity value by applying one or more Jacobi iterations; and
determining the second pixel intensity value by applying a plurality of Jacobi iterations.
6. The method of claim 1 , wherein the information about the gaze direction includes at least one of: a vector gaze direction of the eye, and a distance between a center point of the eye and an origin point perpendicular to a center of a display including at least a portion of the pixels.
7. The method of claim 6 , further comprising:
adjusting the pixel intensity value based on the distance and the vector gaze direction.
8. The method of claim 6 , wherein modifying the second pixel intensity value further comprises:
mapping the second pixel intensity value from a focal plane of a display including at least a portion of the plurality of pixels into another focal plane perpendicular to the vector gaze direction of the eye.
9. The method of claim 1 , further comprising:
determining information about an accommodation of the eye responsive to presenting the content across a plurality of displays including the plurality of pixels, based on the determined pixel intensity value for each pixel of the plurality of pixels; and
modifying the gaze direction for the eye based in part on the determined information about the accommodation.
10. The method of claim 9 , further comprising:
adjusting an optical distance between the eye and at least one display of the plurality of displays, based on the information about the accommodation.
11. The method of claim 1 , further comprising:
instructing a plurality of displays including the plurality of pixels to present the content, based on the determined pixel intensity value for each pixel of the plurality of pixels.
12. The method of claim 1 , wherein the pixels belong to a plurality of displays integrated into a head mounted display (HMD).
13. A computer program product comprising a non-transitory computer-readable storage medium having instructions encoded thereon that, when executed by one or more processors, cause the one or more processors to:
determine a first pixel intensity value for each pixel of a plurality of pixels by applying at least one numerical iteration on a plurality of correlation values related to the plurality of pixels;
determine a second pixel intensity value for each pixel of the plurality of pixels using the first intensity value and applying one or more numerical iterations;
determine information about a gaze direction for an eye;
modify the second pixel intensity value based on the information about the gaze direction; and
determine a pixel intensity value for each pixel of the plurality of pixels to generate content for presentation, based on the modified second pixel intensity value.
14. The computer program product of claim 13 , wherein the instructions further cause the one or more processors to:
obtain a blur gradient correlation value based on the correlation values; and
determining the first pixel intensity value by applying the at least one numerical iteration on the blur gradient correlation value.
15. The computer program product of claim 14 , wherein the instructions further cause the one or more processors to:
subtract first correlation values of the plurality of correlation values in a first kernel corresponding to a first focal distance from second correlation values of the plurality of correlation values in a second kernel corresponding to a second focal distance; and
determine the blur gradient correlation value based on the subtraction.
16. The computer program product of claim 13 , wherein the instructions further cause the one or more processors to:
determine the first pixel intensity value by applying one or more Jacobi iterations; and
determine the second pixel intensity value by applying the one or more Jacobi iterations.
17. The computer program product of claim 13 , wherein the information about the gaze direction includes at least one of: a vector gaze direction of the eye, and a distance between a center point of the eye and an origin point perpendicular to a center of a display including at least a portion of the pixels, and the instructions further cause the one or more processors to:
adjust the pixel intensity value based on the distance and the vector gaze direction.
18. The computer program product of claim 13 , wherein the instructions further cause the one or more processors to:
determine information about an accommodation of the eye responsive to presenting the content across a plurality of displays including the plurality of pixels, based on the determined pixel intensity value for each pixel;
modify the gaze direction for the eye based in part on the determined information about the accommodation; and
adjust an optical distance between the eye and at least one display of the plurality of displays, based on the information about the accommodation.
19. A controller configured to:
determine a first pixel intensity value for each pixel of a plurality of pixels by applying at least one numerical iteration on a plurality of correlation values related to the plurality of pixels;
determine a second pixel intensity value for each pixel of the plurality of pixels using the first intensity value and applying one or more numerical iterations;
determine information about a gaze direction for an eye;
modify the second pixel intensity value based on the information about the gaze direction; and
determine a pixel intensity value for each pixel of the plurality of pixels to generate content for presentation, based on the modified second pixel intensity value.
20. The controller of claim 19 , wherein the controller is further configured to:
obtain a blur gradient correlation value based on the correlation values; and
determine the first pixel intensity value by applying the at least one numerical iteration on the blur gradient correlation value.