Granted Patent B2

US 12,182,945 · App. 17/493,163 · Granted Dec 31, 2024

Inventors: Nicole Elizabeth Samec (Fort Lauderdale, FL); Nastasja U. Robaina (Coconut Grove, FL); Christopher M. Harrises (Nashua, NH); Mark Baerenrodt (Millbrae, CA)

Assignee: MAGIC LEAP, INC.

G06T19/006G02B27/0172G06F3/011G06F3/013G06F3/04842H04N13/122H04N13/128H04N13/332H04N13/383H04N13/395G02B2027/0112G02B2027/0138G02B2027/014G02B2027/0178G02B2027/0181G02B2027/0185G02B2027/0187

Abstract

Claims (63)

1. A system comprising:

a display device comprising a stack of waveguides, the stack of waveguides forming a display area and providing a view of an ambient environment through the display area, wherein a first waveguide of the stack of waveguides comprises diffractive optical elements configured to output light with a first wavefront divergence corresponding to a first accommodation cue and wherein a second waveguide of the stack of waveguides comprises diffractive optical elements configured to output light with a second wavefront divergence corresponding to a second accommodation cue;

one or more processors configured to provide access to information from a map of the ambient environment, the map comprising respective three-dimensional locations for a first virtual object and a second virtual object, wherein a location for the first virtual object is associated with the first wavefront divergence and has associated binocular depth cues, and wherein the second virtual object is associated with the second wavefront divergence and second binocular depth cues; and

one or more computer storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

determining an indication of a fixation point of a user's eyes, wherein the fixation point corresponds to a location of the first virtual object;

obtaining, via the map, location information associated with the second virtual object, wherein the second virtual object is configured to be presented, via the display device, with the second wavefront divergence from the second waveguide and with the second binocular depth cues;

subsequently changing the wavefront divergence associated with the second virtual object to correspond with the first wavefront divergence of the first virtual object, wherein the second binocular depth cues of the second virtual object are retained, wherein changing the wavefront divergence associated with the second virtual object causes an increase in an accommodation vergence mismatch of the second virtual object as long as the accommodation vergence mismatch of the second virtual object is below a threshold value, and

wherein the subsequently changing the wavefront divergence includes adjusting the first accommodation cue based on an indication of a user input for other virtual objects to be adjusted to be at a same depth as the second virtual object,

wherein the subsequently changing the wavefront divergence includes adjusting the first accommodation cue based on an indication of a user input for other virtual objects to be adjusted to be at a same depth as the second virtual object, the user input being independent of the determination of the fixation point of the user's eyes with respect to the first virtual object relative to the second virtual object, and

wherein the user input includes at least one of a hand gesture, a voice input, an interaction with a totem, or a predetermined user preference; and

causing presentation to the user, via the first waveguide of the display device, of the second virtual object with the adjusted second wavefront divergence, wherein, upon determining that the accommodation vergence mismatch exceeds the threshold value, the system is configured to change the second binocular depth cues to reduce the accommodation vergence mismatch to below the threshold value.

2. The system of claim 1 ,

wherein the subsequently changing the wavefront divergence includes adjusting the first accommodation cue further based on a determination that the first virtual object is being fixated on less than the second virtual object, and

wherein determining the indication of the fixation point comprises determining that the user's eyes are fixating on the location for greater than a threshold duration.

3. The system of claim 1 ,

wherein the subsequently changing the wavefront divergence includes adjusting the first accommodation cue further based on a determination that the first virtual object is being fixated on less than the second virtual object, and

wherein subsequently changing the wavefront divergence further comprises determining that the user is alternating fixation, at greater than a threshold frequency, between the first virtual object and the second virtual object.

4. The system of claim 1 , wherein the accommodation-vergence mismatch threshold is 0.33 diopter.

5. The system of claim 1 ,

wherein the display device is a vari-focal display device, and

wherein at least some waveguides of the plurality of waveguides are configured to output light of different component colors than other waveguides.

6. The system of claim 1 , wherein the depth cues further comprise size cues.

7. A display system comprising:

a display device configured to present virtual objects to a user on a plurality of depth planes, wherein each depth plane has a different associated accommodation cue;

one or more processors; and

one or more computer storage media storing instructions that when executed by the display system, cause the display system to perform operations comprising:

accessing a map of the world that comprises respective locations for a first virtual object and a second virtual object, the first virtual object being presented on a first depth plane;

determining an indication of a fixation point of the eyes of the user, the fixation point corresponding to the location of the first virtual object, wherein the determining the indication of the fixation point includes a user input indication of a user input for other virtual objects to be adjusted to be at a same depth as the second virtual object;

the user input being independent of the determination of the fixation point of the user's eyes with respect to the first virtual object relative to the second virtual object, and

wherein the user input includes at least one of a hand gesture, a voice input, an interaction with a totem, or a predetermined user preference; and

based on the determined indication of the fixation point, changing display properties of the second virtual object such that the second virtual object is displayed on the first depth plane,

wherein the second virtual object was configured to be presented on a second depth plane of the plurality of depth planes based on the map, wherein display of the second virtual object is changed to the first depth plane, wherein displaying the second virtual object on the first depth plane causes an increase in an accommodation vergence mismatch of the second virtual object to a threshold value, and wherein the system is configured to change the second binocular depth cues based on a determination that the accommodation vergence mismatch exceeds the threshold value.

8. The display system of claim 7 ,

wherein the determining the indication of the fixation point further includes a determination that the first virtual object is being fixated on less than the second virtual object, and

wherein determining the indication of the fixation point comprises determining that the fixation point falls on the depth plane for greater than a threshold metric before changing display properties of the second virtual object.

9. The display system of claim 7 ,

wherein the determining the indication of the fixation point further includes a determination that the first virtual object is being fixated on less than the second virtual object, and

wherein changing display properties of the second virtual object comprises determining that the second virtual object is being fixated upon less than the first virtual object.

10. The display system of claim 7 , wherein the accommodation-vergence mismatch threshold is 0.33 diopter.

11. The display system of claim 7 , wherein the display device is a vari-focal display device.

12. A method comprising:

by a system comprising one or more processors and a display device comprising a stack of waveguides forming a display area and providing a view of an ambient environment through the display area, the display device configured to present virtual objects to a user on a plurality of depth planes, wherein each depth plane has a different associated accommodation cue:

accessing a map of the world that comprises respective positions for a first virtual object and a second virtual object, the first virtual object being presented via a first waveguide of the stack of waveguides, the first waveguide configured to output light with a first level of wavefront divergence, wherein the first virtual object has associated first binocular depth cues and the second virtual object has an associated second level of wavefront divergence and second binocular depth cues which are different from the first level of wavefront divergence and the first binocular depth cues;

determining an indication of a fixation point of a user's eyes, wherein the fixation point corresponds to a location of the first virtual object;

obtaining, via the map, location information associated with the second virtual object, wherein the second virtual object is configured to be presented, via the display device, with the second level of wavefront divergence and second binocular depth cues, and wherein a second waveguide of the stack of waveguides is configured to present virtual objects with the second accommodation cues;

subsequently changing the wavefront divergence associated with the second virtual object to correspond with the first level of wavefront divergence, wherein the second binocular cues of the second virtual object are retained, wherein changing the wavefront divergence associated with the second virtual object causes an increase in an accommodation vergence mismatch of the second virtual object to a threshold value,

wherein the subsequently changing the wavefront divergence includes adjusting the first accommodation cue based on an indication of a user input for other virtual objects to be adjusted to be at a same depth as the second virtual object,

wherein the subsequently changing the wavefront divergence includes adjusting the first accommodation cue based on an indication of a user input for other virtual objects to be adjusted to be at a same depth as the second virtual object, the user input being independent of the determination of the fixation point of the user's eyes with respect to the first virtual object relative to the second virtual object, and

wherein the user input includes at least one of a hand gesture, a voice input, an interaction with a totem, or a predetermined user preference,

wherein, upon a determination that the accommodation vergence mismatch exceeds the threshold value, changing the second binocular depth cues such that the second virtual object is perceived by the user to be in a common plane as the first virtual object; and

causing presentation to the user, via the first waveguide of the display device, of the second virtual object with the first level of wavefront divergence and the second binocular depth cues.

13. The method of claim 12 ,

wherein the subsequently changing the wavefront divergence includes adjusting the first accommodation cue further based on a determination that the first virtual object is being fixated on less than the second virtual object, and

wherein determining the indication of the fixation point comprises determining that the user's eyes are fixating on the location for greater than a threshold duration.

14. The method of claim 12 ,

wherein the subsequently changing the wavefront divergence includes adjusting the first accommodation cue further based on a determination that the first virtual object is being fixated on less than the second virtual object, and

wherein subsequently changing the wavefront divergence comprises: determining that the user is alternating fixation, at greater than a threshold frequency, between the first virtual object and the second virtual.

15. The method of claim 12 , wherein the display device is a vari-focal display device, and wherein at least some waveguides of the stack of waveguides are configured to output light with different component colors than other waveguide.

16. The system of claim 1 ,

wherein the subsequently changing the wavefront divergence includes adjusting the first accommodation cue further based on a determination that the first virtual object is being fixated on less than the second virtual object, and

wherein the subsequently changing the wavefront divergence includes the determination that the first virtual object is being fixated on less than the second virtual object.

17. The system of claim 1 , wherein the changing the second binocular depth cues upon determining that the accommodation vergence mismatch exceeds the threshold value includes changing the second binocular depth cues such that the second virtual object is perceived by the user to be in a common plane as the first virtual object.

18. The system of claim 1 , wherein the changing the second binocular depth cues upon determining that the accommodation vergence mismatch exceeds the threshold value includes changing the second binocular depth cues such that the second virtual object is perceived by the user to be closer to the user but in a different plane than the first virtual object.

Assignments (5)

Continuity (9)

Cited By (2)

US 12,456,283 US 12,457,318