IP Library Granted Patent US 11,238,836
Granted Patent B2
US 11,238,836 · App. 16/355,603 · Granted Feb 1, 2022

Depth based foveated rendering for display systems

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,238,836
App. No.
16/355,603
Granted
Feb 1, 2022
Kind
B2
Abstract

Methods and systems for depth-based foveated rendering in the display system are disclosed. The display system may be an augmented reality display system configured to provide virtual content on a plurality of depth planes using different wavefront divergence. Some embodiments include determining a fixation point of a user's eyes. Location information associated with a first virtual object to be presented to the user via a display device is obtained. A resolution-modifying parameter of the first virtual object is obtained. A particular resolution at which to render the first virtual object is identified based on the location information and the resolution-modifying parameter of the first virtual object. The particular resolution is based on a resolution distribution specifying resolutions for corresponding distances from the fixation point. The first virtual object rendered at the identified resolution is presented to the user via the display system.

Claims (51)

1. A display system comprising:

a display device configured to output light with a plurality of levels of wavefront divergence, wherein different levels of wavefront divergence correspond to different distances from a fixation point of a user's eyes;

one or more processors; and

one or more computer storage media storing instructions that when executed by the one or more processors, cause the one or more processors to perform operations comprising:

determining the fixation point of the user's eyes;

obtaining location information associated with a first virtual object to be presented to the user via the display device;

obtaining a resolution-modifying parameter of the first virtual object;

identifying, based on the location information and the resolution-modifying parameter of the first virtual object, a particular resolution at which to render the first virtual object, wherein the particular resolution is based on a resolution distribution specifying resolutions for corresponding distances from the fixation point, wherein identifying the particular resolution comprises associating different resolutions with different levels of waterfront divergence, wherein the distances from the fixation point correspond to different angular distances and depths, wherein a field of view of the user is separated into a plurality of portions based on the resolution distribution, wherein each portion encompasses a respective range of angular distances from a center of the field of view, and wherein, for a particular depth or range of depths, each portion is assigned an associated resolution at which to render virtual content; and

causing presentation to the user, via the display device, of the first virtual object rendered at the identified resolution by outputting light from the display device with a level of wavefront divergence associated with the identified resolution.

2. The display system of claim 1 , wherein the resolution-modifying parameter comprises a content type associated with the first virtual object, wherein the operations further comprise:

accessing a plurality of resolution distributions, the plurality of resolution distributions associated with respective content types; and

selecting, from the plurality of resolution distributions, a particular resolution distribution based on the content type of the first virtual object, wherein the particular resolution comprises the particular resolution distribution.

3. The display system of claim 2 , wherein the content type associated with the first virtual object is identified based on a frequency spectrum associated with the first virtual object.

4. The display system of claim 3 , wherein the plurality of resolution distributions are associated with respective rolloffs in resolution away from the fixation point, wherein values of the rolloffs differ for content with different frequency spectrums.

5. The display system of claim 1 , wherein the resolution-modifying parameter is a user selectable value.

6. The display system of claim 5 , wherein the display device is configured to adjust the particular resolution, and wherein adjusting the particular resolution comprises:

causing presentation to the user, via the display device, of a second virtual object, the second virtual object being rendered with the resolution distribution identified for the first virtual object;

receiving, from the user, a response indicating user detection of reduction in resolution of the second virtual object, wherein the user response is the user selectable value; and

adjusting the particular resolution distribution.

7. The display system of claim 6 , wherein adjusting the particular resolution distribution comprises:

adjusting a rolloff associated with the particular resolution distribution, wherein adjusting a rolloff changes an amount of resolution reduction based on angular distance from the center of the field of view of the user.

8. The display system of claim 1 , wherein the fixation point is in a volume at the center of the field of view of the user.

9. The display system of claim 1 , wherein the operations further comprise:

determining a proximity of the first virtual object to a boundary of one of the plurality of portions; and

modifying a presentation of the first virtual objected based on the determined proximity.

10. The display system of claim 9 , wherein modifying a presentation of the first virtual objected based on the determined proximity comprises applying a blurring process to the virtual object.

11. The display system of claim 1 , wherein identifying a particular resolution at which to render the first virtual object comprises:

identifying a portion of the plurality of portions which encompasses the first virtual object; and

identifying the resolution based on the portion which encompasses the first virtual object.

12. A computer-implemented method, the method being performed by a display system of one or more processors, and the method comprising:

determining a fixation point of a user's eyes;

obtaining location information associated with a first virtual object to be presented to the user via a display device configured to output light with a plurality of levels of wavefront divergence, wherein different levels of wavefront divergence correspond to different distances from the fixation point of the user's eyes;

obtaining a resolution-modifying parameter of the first virtual object;

identifying, based on the location information and the resolution-modifying parameter of the first virtual object, a particular resolution at which to render the first virtual object, wherein the particular resolution is based on a resolution distribution specifying resolutions for corresponding distances from the fixation point, wherein identifying the particular resolution comprises associating different resolutions with different levels of wavefront divergence, wherein the distances from the fixation point correspond to different angular distances and depths, wherein a field of view of the user is separated into a plurality of portions based on the resolution distribution, wherein each portion encompasses a respective range of angular distances from a center of the field of view, and wherein, for a particular depth or range of depths, each portion is assigned an associated resolution at which to render virtual content; and

causing presentation to the user, via the display device, of the first virtual object rendered at the identified resolution by outputting light from the display device with a level of wavefront divergence associated with the identified resolution.

13. The computer-implemented method of claim 12 , wherein the resolution-modifying parameter comprises a content type associated with the first virtual object, wherein the method further comprises:

accessing a plurality of resolution distributions, the plurality of resolution distributions associated with respective content types; and

selecting, from the plurality of resolution distributions, a particular resolution distribution based on the content type of the first virtual object, wherein the particular resolution comprises the particular resolution distribution.

14. The computer-implemented method of claim 13 , wherein the content type associated with the first virtual object is identified based on a frequency spectrum associated with the first virtual object.

15. The computer-implemented method of claim 13 , wherein the plurality of resolution distributions are associated with respective rolloffs in resolution away from the fixation point, wherein values of the rolloffs differ for content with different frequency spectrums.

16. Non-transitory computer storage media storing instructions that when executed by a display system of one or more processors, cause the one or more processors to perform operations comprising:

determining a fixation point of a user's eyes;

obtaining location information associated with a first virtual object to be presented to the user via a display device configured to output light with a plurality of levels of wavefront divergence, wherein different levels of wavefront divergence correspond to different distances from the fixation point of the user's eyes;

obtaining a resolution-modifying parameter of the first virtual object;

identifying, based on the location information and the resolution-modifying parameter of the first virtual object, a particular resolution at which to render the first virtual object, wherein the particular resolution is based on a resolution distribution specifying resolutions for corresponding distances from the fixation point, wherein identifying the particular resolution comprises associating different resolutions with different levels of wavefront divergence, wherein the distances from the fixation point correspond to different angular distances and depths, wherein a field of view of the user is separated into a plurality of portions based on the resolution distribution, wherein each portion encompasses a respective range of angular distances from a center of the field of view, and wherein, for a particular depth or range of depths, each portion is assigned an associated resolution at which to render virtual content; and

causing presentation to the user, via the display device, of the first virtual object rendered at the identified resolution by outputting light from the display device with a level of wavefront divergence associated with the identified resolution.

17. The non-transitory computer storage media of claim 16 , wherein the resolution-modifying parameter comprises a content type associated with the first virtual object, wherein the operations further comprise:

accessing a plurality of resolution distributions, the plurality of resolution distributions associated with respective content types; and

selecting, from the plurality of resolution distributions, a particular resolution distribution based on the content type of the first virtual object, wherein the particular resolution comprises the particular resolution distribution.

18. The non-transitory computer storage media of claim 17 , wherein the content type associated with the first virtual object is identified based on a frequency spectrum associated with the first virtual object.

19. The non-transitory computer storage media of claim 17 , wherein the plurality of resolution distributions are associated with respective rolloffs in resolution away from the fixation point, wherein values of the rolloffs differ for content with different frequency spectrums.

Assignments (5)
SECURITY INTEREST Recorded Oct 28, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073388/0027 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 12, 2021
From: MATHUR, VAIBHAV; EDWIN, LIONEL ERNEST; VLASKAMP, BJORN NICOLAAS SERVATIUS
To: MAGIC LEAP, INC.
Reel/Frame 055250/0745 →
PROPRIETARY INFORMATION AND INVENTIONS AGREEMENT Recorded Feb 12, 2021
From: ZHANG, XIAOYANG
To: MAGIC LEAP, INC.
Reel/Frame 055293/0001 →
ASSIGNMENT OF SECURITY INTEREST IN PATENTS Recorded Nov 7, 2019
From: JPMORGAN CHASE BANK, N.A.
To: CITIBANK, N.A.
Reel/Frame 050967/0138 →
PATENT SECURITY AGREEMENT Recorded Aug 22, 2019
From: MAGIC LEAP, INC.; MOLECULAR IMPRINTS, INC.; MENTOR ACQUISITION ONE, LLC
To: JP MORGAN CHASE BANK, N.A.
Reel/Frame 050138/0287 →