Real-time multifocal displays with gaze-contingent rendering and optimization

1. A method comprising:

determining a plurality of correlation values related to a plurality of pixels of a plurality of displays;

initializing a pixel intensity value for each pixel in at least a portion of the plurality of pixels to a defined intensity value;

determining a first pixel intensity value for each pixel in at least the portion of the plurality of pixels in an iterative manner by modifying the defined intensity value using at least one numerical iteration applied on blur gradient correlation values obtained based on the plurality of correlation values;

determining a second pixel intensity value for each pixel in at least the portion of the plurality of pixels in an iterative manner using the first intensity value and applying at least one numerical iteration;

determining, using an eye tracking system, eye tracking information about a position of the eye relative to the plurality of displays;

modifying the second pixel intensity value for each pixel in at least the portion of the plurality of pixels based on the eye tracking information to determine a pixel intensity value for each pixel in at least the portion of the plurality of pixels; and

displaying an image across the plurality of displays based on the determined pixel intensity value for each pixel in at least the portion of the plurality of pixels.

2. The method of claim 1 , wherein at least two of the plurality of displays are allocated to each eye channel associated with an eye, the at least two displays located at a different focal distance from one another.

3. The method of claim 1 , wherein determining the first pixel intensity value comprises:

determining a difference between a first correlation value corresponding to a first focal distance and a second correlation value corresponding to a second focal distance; and

determining a blur gradient correlation value of the blur gradient correlation values based on the difference.

4. The method of claim 1 , wherein the at least one numerical iteration includes one or more Jacobi iterations.

5. The method of claim 1 , wherein the at least one numerical iteration is one Jacobi iteration.

6. The method of claim 1 , wherein the second pixel intensity value is determined by applying a plurality of Jacobi iterations.

7. The method of claim 1 , wherein the information about position of the eye includes a vector gaze direction of the eye.

8. The method of claim 7 , wherein modifying the second pixel intensity value further comprises:

applying a mapping transformation to the second pixel intensity value for each pixel in at least the portion of the plurality of pixels, wherein the mapping transformation maps from a focal plane of a display in the plurality of displays to a focal plane perpendicular to the vector gaze direction of the eye.

9. The method of claim 1 , wherein the plurality of displays are integrated into a head mounted display (HMD).

10. The method of claim 1 , further comprising:

determining information about an accommodation of the eye responsive to displaying the image across the plurality of displays based on the determined pixel intensity value for each pixel in at least the portion of the plurality of pixels; and

modifying the information about the position of the eye, based in part on the determined information about the accommodation of the eye.

11. A non-transitory computer readable storage medium storing processor executable instructions, the instructions comprising instructions for:

determining a plurality of correlation values related to a plurality of pixels of a plurality of displays;

initializing a pixel intensity value for each pixel in at least a portion of the plurality of pixels to a defined intensity value;

determining a first pixel intensity value for each pixel in at least the portion of the plurality of pixels in an iterative manner by modifying the defined intensity value using at least one numerical iteration applied on blur gradient correlation values obtained based on the plurality of correlation values;

determining a second pixel intensity value for each pixel in at least the portion of the plurality of pixels in an iterative manner using the first intensity value and applying at least one numerical iteration;

determining, using an eye tracking system, eye tracking information about a position of the eye relative to the plurality of displays;

modifying the second pixel intensity value for each pixel in at least the portion of the plurality of pixels based on the eye tracking information to determine a pixel intensity value for each pixel in at least the portion of the plurality of pixels; and

displaying an image across the plurality of displays based on the determined pixel intensity value for each pixel in at least the portion of the plurality of pixels.

12. The computer readable medium of claim 11 , wherein the instructions further comprising instructions for:

determining a difference between a first correlation value corresponding to a first focal distance associated with a first display of the plurality of displays and a second correlation value corresponding to a second focal distance associated with a second display of the plurality of displays; and

determining a blur gradient correlation value of the blur gradient correlation values based on the difference.

13. The computer readable medium of claim 11 , wherein the at least one numerical iteration includes one or more Jacobi iterations.

14. The computer readable medium of claim 11 , wherein the at least one numerical iteration is one Jacobi iteration.

15. The computer readable medium of claim 11 , wherein the second pixel intensity value is determined by applying a plurality of Jacobi iterations.

16. The computer readable medium of claim 11 , wherein the information about the position of the eye includes a vector gaze direction of the eye.

17. The computer readable medium of claim 16 , wherein the instructions further comprising instructions for:

applying a mapping transformation to the second pixel intensity value for each pixel in at least the portion of the plurality of pixels, wherein the mapping transformation maps from a focal plane of a display in the plurality of displays to a focal plane perpendicular to the vector gaze direction of the eye.

18. The computer readable medium of claim 11 , wherein the instructions further comprising instructions for:

determining information about an accommodation of the eye responsive to displaying the image across the plurality of displays based on the determined pixel intensity value for each pixel in at least the portion of the plurality of pixels; and

modifying the eye tracking information, based in part on the determined information about the accommodation of the eye.

19. A controller configured to:

determine a plurality of correlation values related to a plurality of pixels of a plurality of displays;

initialize a pixel intensity value for each pixel in at least a portion of the plurality of pixels to a defined intensity value;

determine a first pixel intensity value for each pixel in at least the portion of the plurality of pixels in an iterative manner by modifying the defined intensity value using at least one numerical iteration applied on blur gradient correlation values obtained based on the plurality of correlation values;

determine a second pixel intensity value for each pixel in at least the portion of the plurality of pixels in an iterative manner using the first intensity value and applying at least one numerical iteration;

receive eye tracking information about a position of the eye relative to the plurality of displays;

modify the second pixel intensity value for each pixel in at least the portion of the plurality of pixels based on the position of the eye to determine a pixel intensity value for each pixel in at least the portion of the plurality of pixels; and

instruct the plurality of displays to display an image across the plurality of displays based on the determined pixel intensity value for each pixel in at least the portion of the plurality of pixels.

20. The controller of claim 19 , wherein the controller is further configured to:

receive information about an accommodation of the eye responsive to displaying the image across the plurality of displays based on the determined pixel intensity value for each pixel in at least the portion of the plurality of pixels; and

modify the eye tracking information, based in part on the received information about the accommodation of the eye.