IP Library Granted Patent US 11,561,615
Granted Patent B2
US 11,561,615 · App. 17/886,323 · Granted Jan 24, 2023

Multimodal eye tracking

Inventors: Edwin Joseph Selker (Palo Alto, CA); Ivan Yeoh (Fort Lauderdale, FL)
Assignee: Magic Leap, Inc.
G06F3/013G02B27/017G06F3/011G06F3/0346G06T19/006
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,561,615
App. No.
17/886,323
Granted
Jan 24, 2023
Kind
B2
Abstract

A method is disclosed, the method comprising the steps of receiving, at a first time interval from a first sensor configured to output data indicative of a first position of an eye, first data; receiving, at a second time interval from a second sensor configured to output data indicative of a delta position of the eye, second data; determining, based on the first data, a first position of the eye; determining, based on the second data, a delta position of the eye; determining, using the first position of the eye and the delta position of the eye, a second absolute position of the eye; and in response to determining the second position of the eye, generating an output signal indicative of the second position of the eye.

Claims (72)

1. A method comprising:

receiving first data indicative of a first position of an eye, the first data received at a first rate from a first sensor;

receiving second data indicative of a second position of the eye, the second data received at the first rate from the first sensor;

determining, based on the first data and on the second data, whether a displacement of the eye exceeds a threshold;

in accordance with a determination that the displacement of the eye exceeds the threshold:

capturing an image of the eye via a second sensor distinct from the first sensor; and

determining, based on the captured image, an eye gaze of the eye; and

in accordance with a determination that the displacement of the eye does not exceed the threshold:

forgoing capturing the image of the eye via the second sensor.

2. The method of claim 1 , wherein the first sensor comprises an electrooculography sensor.

3. The method of claim 1 , wherein the second sensor comprises an optical sensor.

4. The method of claim 1 , wherein the second sensor operates in a low-power mode concurrently with the receiving of the first data at the first rate.

5. The method of claim 1 , wherein capturing the image of the eye via the second sensor comprises capturing an image of a plurality of images, the plurality of images detected at a second rate via the second sensor.

6. The method of claim 5 , further comprising:

determining, based on an optical sensor parameter, whether the first rate is to be increased with respect to the second rate;

in accordance with a determination that the first rate is to be increased with respect to the second rate, adjusting a system parameter to increase the first rate with respect to the second rate; and

in accordance with a determination that the first rate is to not be increased with respect to the second rate, forgoing adjusting the system parameter.

7. The method of claim 6 , wherein the determination that the first rate is to be increased with respect to the second rate comprises receiving an indication to reduce a power consumption of the second sensor.

8. The method of claim 1 , further comprising:

determining, based on the eye gaze, a first eye movement behavior.

9. The method of claim 8 , wherein determining the first eye movement behavior comprises:

generating a confidence score corresponding to a likelihood of the first eye movement behavior;

comparing the confidence score to a threshold value; and

determining that the confidence score exceeds the threshold value.

10. The method of claim 1 , wherein:

the first sensor and the second sensor are associated with a head-mounted device comprising a see-through display, and

the method further comprises:

determining a region of the display corresponding to a target of the eye gaze, the region having a display state equal to a first display state; and

changing the display state of the region from the first display state to a second display state.

11. A system comprising:

a head-mounted device comprising:

a see-through display;

a first sensor configured to detect eye data; and

a second sensor configured to detect image data;

one or more processors operatively coupled to the first sensor and the second sensor, the one or more processors configured to perform a method comprising:

receiving, at a first rate via the first sensor, first data indicative of a first position of an eye;

receiving, at the first rate via the first sensor, second data indicative of a second position of the eye;

determining, based on the first data and on the second data, whether a displacement of the eye exceeds a threshold;

in accordance with a determination that the displacement of the eye exceeds the threshold:

capturing an image of the eye via the second sensor; and

determining, based on the captured image, an eye gaze of the eye; and

in accordance with a determination that the displacement of the eye does not exceed the threshold:

forgoing capturing the image of the eye via the second sensor.

12. The system of claim 11 , wherein the first sensor comprises an electrooculography sensor and the second sensor comprises an optical sensor.

13. The system of claim 11 , wherein capturing the image of the eye via the second sensor comprises capturing an image of a plurality of images, the plurality of images detected at a second rate via the second sensor.

14. The system of claim 13 , wherein the method further comprises:

determining, based on an optical sensor parameter, whether the first rate is to be increased with respect to the second rate;

in accordance with a determination that the first rate is to be increased with respect to the second rate, adjusting a system parameter to increase the first rate with respect to the second rate; and

in accordance with a determination that the first rate is to not be increased with respect to the second rate, forgoing adjusting the system parameter.

15. The system of claim 11 , wherein the method further comprises:

determining a region of the see-through display corresponding to a target of the eye gaze, the region having a display state equal to a first display state; and

changing the display state of the region from the first display state to a second display state.

16. A non-transitory computer-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform a method comprising:

receiving first data indicative of a first position of an eye, the first data received at a first rate from a first sensor;

receiving second data indicative of a second position of the eye, the second data received at the first rate from the first sensor;

determining, based on the first data and on the second data, whether a displacement of the eye exceeds a threshold;

in accordance with a determination that the displacement of the eye exceeds the threshold:

capturing an image of the eye via a second sensor distinct from the first sensor; and

determining, based on the captured image, an eye gaze of the eye; and

in accordance with a determination that the displacement of the eye does not exceed the threshold:

forgoing capturing the image of the eye via the second sensor.

17. The non-transitory computer-readable medium of claim 16 , wherein the first sensor comprises an electrooculography sensor and the second sensor comprises an optical sensor.

18. The non-transitory computer-readable medium of claim 16 , wherein capturing the image of the eye via the second sensor comprises capturing an image of a plurality of images, the plurality of images detected at a second rate via the second sensor.

19. The non-transitory computer-readable medium of claim 18 , wherein the method further comprises:

determining, based on an optical sensor parameter, whether the first rate is to be increased with respect to the second rate;

in accordance with a determination that the first rate is to be increased with respect to the second rate, adjusting a system parameter to increase the first rate with respect to the second rate; and

in accordance with a determination that the first rate is to not be increased with respect to the second rate, forgoing adjusting the system parameter.

20. The non-transitory computer-readable medium of claim 16 , wherein:

the first sensor and the second sensor are associated with a head-mounted device comprising a see-through display, and

the method further comprises:

determining a region of the see-through display corresponding to a target of the eye gaze, the region having a display state equal to a first display state; and

changing the display state of the region from the first display state to a second display state.

Assignments (3)
SECURITY INTEREST Recorded Oct 15, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073109/0476 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 12, 2023
From: SELKER, EDWIN JOSEPH; YEOH, IVAN
To: MAGIC LEAP, INC.
Reel/Frame 064233/0356 →
SECURITY INTEREST Recorded Feb 7, 2023
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 062681/0065 →
Continuity (5)
Continuation 17678314 · Feb 23, 2022
Continuation 17239451 · Apr 23, 2021
Continuation 15953432 · Apr 14, 2018
Provisional Application 62485820 · Apr 14, 2017
Related Publication 20220382371A1 · Dec 1, 2022