IP Library Granted Patent US 11,710,469
Granted Patent B2
US 11,710,469 · App. 17/581,760 · Granted Jul 25, 2023

Depth based foveated rendering for display systems

Inventors: Vaibhav Mathur (Weston, FL); Lionel Ernest Edwin (Hollywood, FL); Xiaoyang Zhang (Alviso, CA); Bjorn Nicolaas Servatius Vlaskamp (Plantation, FL)
Assignee: Magic Leap, Inc.
G09G5/391G06F3/013G06T5/002G09G2320/0606G09G2320/0613G09G2340/0428G09G2354/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,710,469
App. No.
17/581,760
Granted
Jul 25, 2023
Kind
B2
Abstract

Methods and systems for depth-based foveated rendering in the display system are disclosed. The display system may be an augmented reality display system configured to provide virtual content on a plurality of depth planes using different wavefront divergence. Some embodiments include determining a fixation point of a user's eyes. Location information associated with a first virtual object to be presented to the user via a display device is obtained. A resolution-modifying parameter of the first virtual object is obtained. A particular resolution at which to render the first virtual object is identified based on the location information and the resolution-modifying parameter of the first virtual object. The particular resolution is based on a resolution distribution specifying resolutions for corresponding distances from the fixation point. The first virtual object rendered at the identified resolution is presented to the user via the display system.

Claims (52)

1. A head-mounted display system comprising:

a display configured to output light with a plurality of levels of wavefront divergence, wherein different levels of wavefront divergence correspond to different distances from a fixation point of a user's eyes;

one or more processors; and

one or more computer storage media storing instructions that when executed by the one or more processors, cause the one or more processors to perform operations comprising:

determining the fixation point of the user's eyes;

obtaining location information associated with a first virtual object to be presented to the user via the display;

obtaining a resolution-modifying parameter of the first virtual object;

identifying, based on the location information and the resolution-modifying parameter of the first virtual object, a particular resolution at which to render the first virtual object, wherein the particular resolution is based on a resolution distribution specifying resolutions for corresponding distances from the fixation point, wherein identifying the particular resolution comprises associating different resolutions with different levels of wavefront divergence, wherein a field of view of the user is separated into a plurality of portions based on the resolution distribution, and wherein, for a particular depth or range of depths, each portion is assigned an associated resolution at which to render virtual content; and

causing presentation to the user, via the display, of the first virtual object rendered at the identified resolution by outputting light from the display with a level of wavefront divergence associated with the identified resolution.

2. The head-mounted display system of claim 1 , wherein the resolution-modifying parameter comprises a content type associated with the first virtual object, wherein the operations further comprise:

accessing a plurality of resolution distributions, the plurality of resolution distributions associated with respective content types; and

selecting, from the plurality of resolution distributions, a particular resolution distribution based on the content type of the first virtual object, wherein the particular resolution comprises the particular resolution distribution.

3. The head-mounted display system of claim 2 , wherein the content type associated with the first virtual object is identified based on a frequency spectrum associated with the first virtual object.

4. The head-mounted display system of claim 3 , wherein the plurality of resolution distributions are associated with respective rolloffs in resolution away from the fixation point, wherein values of the rolloffs differ for content with different frequency spectrums.

5. The head-mounted display system of claim 1 , wherein the resolution-modifying parameter is a user selectable value.

6. The head-mounted display system of claim 5 , wherein the display is configured to adjust the particular resolution, and wherein adjusting the particular resolution comprises:

causing presentation to the user, via the display, of a second virtual object, the second virtual object being rendered with the resolution distribution identified for the first virtual object;

receiving, from the user, a response indicating user detection of reduction in resolution of the second virtual object, wherein the user response is the user selectable value; and

adjusting the particular resolution distribution.

7. The head-mounted display system of claim 6 , wherein adjusting the particular resolution distribution comprises:

adjusting a rolloff associated with the particular resolution distribution, wherein adjusting a rolloff changes an amount of resolution reduction based on a distance from the center of the field of view of the user.

8. The head-mounted display system of claim 1 , wherein the fixation point is in a volume at the center of the field of view of the user.

9. The head-mounted display system of claim 1 , wherein the operations further comprise:

determining a proximity of the first virtual object to a boundary of one of the plurality of portions; and

modifying a presentation of the first virtual objected based on the determined proximity.

10. The head-mounted display system of claim 1 , wherein modifying a presentation of the first virtual objected based on the determined proximity comprises applying a blurring process to the virtual object.

11. The head-mounted display system of claim 1 , wherein identifying the particular resolution at which to render the first virtual object comprises:

identifying a portion of the plurality of portions which encompasses the first virtual object; and

identifying the resolution based on the portion which encompasses the first virtual object.

12. A computer-implemented method, the method being performed by a head-mounted display system comprising one or more processors, and the method comprising:

determining a fixation point of a user's eyes;

obtaining location information associated with a first virtual object to be presented to the user via a display configured to output light with a plurality of levels of wavefront divergence, wherein different levels of wavefront divergence correspond to different distances from the fixation point of the user's eyes;

obtaining a resolution-modifying parameter of the first virtual object;

identifying, based on the location information and the resolution-modifying parameter of the first virtual object, a particular resolution at which to render the first virtual object, wherein the particular resolution is based on a resolution distribution specifying resolutions for corresponding distances from the fixation point, wherein identifying the particular resolution comprises associating different resolutions with different levels of wavefront divergence, wherein a field of view of the user is separated into a plurality of portions based on the resolution distribution, and wherein, for a particular depth or range of depths, each portion is assigned an associated resolution at which to render virtual content; and

causing presentation to the user, via the display, of the first virtual object rendered at the identified resolution by outputting light from the display with a level of wavefront divergence associated with the identified resolution.

13. The computer-implemented method of claim 12 , wherein the resolution-modifying parameter comprises a content type associated with the first virtual object, wherein the method further comprises:

accessing a plurality of resolution distributions, the plurality of resolution distributions associated with respective content types; and

selecting, from the plurality of resolution distributions, a particular resolution distribution based on the content type of the first virtual object, wherein the particular resolution comprises the particular resolution distribution.

14. The computer-implemented method of claim 13 , wherein the content type associated with the first virtual object is identified based on a frequency spectrum associated with the first virtual object.

15. The computer-implemented method of claim 13 , wherein the plurality of resolution distributions are associated with respective rolloffs in resolution away from the fixation point, wherein values of the rolloffs differ for content with different frequency spectrums.

16. The computer-implemented method of claim 13 , wherein the resolution-modifying parameter is a user selectable value.

17. Non-transitory computer storage media storing instructions that when executed by a head-mounted display system of one or more processors, cause the one or more processors to perform operations comprising:

determining a fixation point of a user's eyes;

obtaining location information associated with a first virtual object to be presented to the user via a display configured to output light with a plurality of levels of wavefront divergence, wherein different levels of wavefront divergence correspond to different distances from the fixation point of the user's eyes;

obtaining a resolution-modifying parameter of the first virtual object;

identifying, based on the location information and the resolution-modifying parameter of the first virtual object, a particular resolution at which to render the first virtual object, wherein the particular resolution is based on a resolution distribution specifying resolutions for corresponding distances from the fixation point, wherein identifying the particular resolution comprises associating different resolutions with different levels of wavefront divergence, wherein a field of view of the user is separated into a plurality of portions based on the resolution distribution, and wherein, for a particular depth or range of depths, each portion is assigned an associated resolution at which to render virtual content; and

causing presentation to the user, via the display, of the first virtual object rendered at the identified resolution by outputting light from the display with a level of wavefront divergence associated with the identified resolution.

18. The non-transitory computer storage media of claim 17 , wherein the resolution-modifying parameter comprises a content type associated with the first virtual object, wherein the operations further comprise:

accessing a plurality of resolution distributions, the plurality of resolution distributions associated with respective content types; and

selecting, from the plurality of resolution distributions, a particular resolution distribution based on the content type of the first virtual object, wherein the particular resolution comprises the particular resolution distribution.

19. The non-transitory computer storage media of claim 18 , wherein the content type associated with the first virtual object is identified based on a frequency spectrum associated with the first virtual object.

20. The non-transitory computer storage media of claim 18 , wherein the plurality of resolution distributions are associated with respective rolloffs in resolution away from the fixation point, wherein values of the rolloffs differ for content with different frequency spectrums.

Assignments (4)
SECURITY INTEREST Recorded Oct 20, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073008/0696 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 15, 2023
From: MATHUR, VAIBHAV; EDWIN, LIONEL ERNEST; VLASKAMP, BJORN NICOLAAS SERVATIUS
To: MAGIC LEAP, INC.
Reel/Frame 062986/0335 →
PROPRIETARY INFORMATION AND INVENTIONS AGREEMENT Recorded Mar 15, 2023
From: ZHANG, XIAOYANG
To: MAGIC LEAP, INC.
Reel/Frame 063099/0710 →
SECURITY INTEREST Recorded May 24, 2022
From: MOLECULAR IMPRINTS, INC.; MENTOR ACQUISITION ONE, LLC; MAGIC LEAP, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 060338/0665 →
Continuity (3)
Continuation 16355603 · Mar 15, 2019
Provisional Application 62644366 · Mar 16, 2018
Related Publication 20220148538A1 · May 12, 2022