Rendering for multi-focus display systems
Some implementations provide a multi-focus display system that renders images at multiple focus distances for display in conjunction with the use of appropriately powered lenses. For example, an HMD may include a fast switching lens element that allows quickly alternating between two or more focus distances. The displayed images are configured to correspond to the alternating focus distances by adjusting a high-frequency part of the images. This can provide a more natural user experience that will include near objects that require the user's eye to focus on a close focal depth plane and far objects that require the user's eye to focus on a far focal depth plane. Moreover, the user experience can be provided with little or no loss of brightness and without requiring processor and resource intensive computations.
1. A method, comprising:
at an electronic device having a processor:
obtaining a sequence of images of a scene;
determining portions of the images associated with each of multiple focal depth planes;
creating adjusted images by adjusting a high-frequency part of each portion based on which of the multiple focal depth planes is associated with the respective portion; and
providing the adjusted images for sequential display on a head-mounted device (HMD).
2. The method of claim 1 , wherein creating the adjusted images comprises:
creating images of a first focus type in which first portions associated with a first focal depth plane are in focus and second portions associated with a second focal depth plane are out of focus; and
creating images of a second focus type in which second portions associated with the second focal depth plane are in focus and first portions associated with the first focal depth plane are out of focus.
3. The method of claim 2 , wherein creating the adjusted images comprises:
for images of the first focus type, boosting the high-frequency part of the first portions and reducing the high-frequency part of the second portions; and
for images of the second focus type, boosting the high-frequency part of the second portions and reducing the high-frequency part of the first portions.
4. The method of claim 2 , wherein creating the adjusted images comprising creating a sequence of alternating adjusted images of the first focal type and adjusted images of the second focal type.
5. The method of claim 2 , wherein creating the adjusted images comprises:
creating images of a third focus type in which third portions associated with a third focal depth plane are in focus and the first portions and the second portions are out of focus.
6. The method of claim 5 , wherein creating the adjusted images comprises
for images of the first focus type, boosting the high-frequency part of the first portions and reducing the high-frequency part of the second portions and the third portions;
for images of the second focus type, boosting the high-frequency part of the second portions and reducing the high-frequency part of the first portions and the third portions; and
for images of the third focus type, boosting the high-frequency part of the third portions and reducing the high-frequency part of the first portions and the second portions.
7. The method of claim 5 , wherein creating the adjusted images comprising creating a sequence of alternating adjusted images of the first focal type, adjusted images of the second focal type, and adjusted images of the third focal type.
8. The method of claim 1 , wherein determining the portions of the images associated with each of the multiple focal depth planes comprises:
determining portions of the images associated with a foreground focal depth plane; and
determining portions of the images associated with a background focal depth plane.
9. The method of claim 1 , wherein determining the portions of the images associated with each of the multiple focal depth planes comprises:
determining a mask based on depth information for the images.
10. The method of claim 1 , wherein determining the portions of the images associated with each of the multiple focal depth planes comprises:
obtaining depth information identifying depth of objects corresponding to pixels of the images;
identifying a distribution of pixel depths; and
determining the multiple focal depth planes based on the distribution of pixel depths and focal depth plane selection criteria.
11. The method of claim 1 further comprising:
detecting a current focus of an eye; and
compensating for naturally-occurring blur of the eye associated with a first depth plane on content associated with another depth plane.
12. The method of claim 1 further comprising:
determining to change a segmentation of the multiple focal depth planes;
detecting a blink or saccade; and
changing the segmentation of the multiple focal depth planes during the blink or saccade.
13. A system comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
a sequence of images of a scene;
determining portions of the images associated with each of multiple focal depth planes;
creating adjusted images by adjusting a high-frequency part of each portion based on which of the multiple focal depth planes is associated with the respective portion; and
providing the adjusted images for sequential display on a head-mounted device (HMD).
14. The system of claim 13 , wherein creating the adjusted images comprises:
creating images of a first focus type in which first portions associated with a first focal depth plane are in focus and second portions associated with a second focal depth plane are out of focus; and
creating images of a second focus type in which first portions associated with the second focal depth plane are in focus and second portions associated with the first focal depth plane are out of focus.
15. The system of claim 14 , wherein creating the adjusted images comprises:
for images of the first focus type, boosting the high-frequency part of the first portions and reducing the high-frequency part of the second portions; and
for images of the second focus type, boosting the high-frequency part of the second portions and reducing the high-frequency part of the first portions.
16. The system of claim 14 , wherein creating the adjusted images comprising creating a sequence of alternating adjusted images of the first focal type and adjusted images of the second focal type.
17. The system of claim 14 , wherein creating the adjusted images comprises:
creating images of a third focus type in which third portions associated with a third focal depth plane are in focus and the first portions and the second portions are out of focus.
18. The system of claim 17 , wherein creating the adjusted images comprises:
for images of the first focus type, boosting the high-frequency part of the first portions and reducing the high-frequency part of the second portions and the third portions;
for images of the second focus type, boosting the high-frequency part of the second portions and reducing the high-frequency part of the first portions and the third portions; and
for images of the third focus type, boosting the high-frequency part of the third portions and reducing the high-frequency part of the first portions and the second portions.
19. The system of claim 17 , wherein creating the adjusted images comprising creating a sequence of alternating adjusted images of the first focal type, adjusted images of the second focal type, and adjusted images of the third focal type.
20. A method, comprising:
at an electronic device having a processor:
obtaining a sequence of images of a scene;
obtaining depth information identifying depth of objects corresponding to pixels of the images;
identifying a distribution of pixel depths based on the depth information, the pixel depths corresponding to depth of individual pixels of the images;
determining multiple focal depth planes based on the distribution of pixel depths and focal depth plane selection criteria;
determining portions of the images associated with each of multiple focal depth planes;
creating adjusted images based on which of the multiple focal depth planes is associated with the portions; and
providing the adjusted images for sequential display on a head-mounted device (HMD).
21. The method of claim 20 , wherein determining portions of the images associated with each of multiple focal depth planes comprises:
determining a threshold based on the distribution of pixel depths; and
determining the portions of the images associated with each of multiple focal depth planes based on the threshold.