IP Library Granted Patent US 11,966,055
Granted Patent B2
US 11,966,055 · App. 17/256,974 · Granted Apr 23, 2024

Content interaction driven by eye metrics

Inventors: Heather Michelle Martin (Fort Lauderdale, FL); Kevin John O'Brien (Plantation, FL); Pedro Luis Arroyo (Fort Lauderdale, FL); Flavio DeOliveira (Miami, FL)
Assignee: Magic Leap, Inc.
G02B27/0172G02B27/0093G02B2027/0127G02B2027/0138G02B2027/014G02B2027/0178
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,966,055
App. No.
17/256,974
Granted
Apr 23, 2024
Kind
B2
Abstract

A head mounted display system for displaying image content to a user comprises at least one display configured to be worn by a user to present virtual content to first and second eyes of a user, one or more inwardly facing sensors or camera configured to monitor one or both of the users eye and processing electronics. This head mounted display system is configured such that virtual content activity can be initiated and/or driven from eye inputs such as gaze direction, eyelid motions (e.g., blinking), and/or other eye gestures.

Claims (28)

1. A head mounted display system for displaying image content to a user, the display system comprising:

at least one display configured to be worn by a user to present virtual content to first and second eyes of a user;

a first inward facing camera configured to image the first eye of the user;

a second inward facing camera configured to image the second eye of the user; and

processing electronics in communication with the first and second inward-facing cameras and the at least one display, the processing electronics configured to:

present virtual content to the user on the at least one display;

using the first inward-facing camera, determine that the first eye is at least partially closed;

using the second inward-facing camera, determine that the second eye is at least partially closed;

determine a gaze direction of the first and second eyes;

determine a vergence of the first and second eyes; and

based on (i) the determination that the first and second eyes of the user are at least partially closed, (ii) the gaze direction of the first and second eyes, and (iii) the vergence of the first and second eyes, alter a state of the head mounted display system so as to modify the virtual content presented to the user, a manner in which virtual content is presented to the user, a performance feature of the head mounted display system apparent to the user, or any combination thereof.

2. The display system of claim 1 , wherein the processing electronics are configured to alter said state of the head mounted display system based on a determination that the first and second eyes of the user are at least partially closed for longer than a threshold time.

3. The display system of claim 2 , wherein said threshold time is in the range from 500 milliseconds to 1.5 seconds.

4. The display system of claim 2 , wherein said threshold time in the range from 750 milliseconds to 2.0 seconds.

5. The display system of claim 2 , wherein said threshold time in the range from 1 second and 3 seconds.

6. The display system of claim 2 , wherein said threshold time in the range from 3 second and 10 seconds.

7. The display system of claim 1 , wherein the processing electronics are configured to alter said state of the head mounted display system based on a determination that the first and second eyes of the user transition from being in an opened state to being in an at least partially closed state at least a first time and a second time.

8. The display system of claim 7 , wherein said first and second times are separated by less than 3 seconds.

9. The display system of claim 7 , wherein said first and second times are separated by less than 2 seconds.

10. The display system of claim 7 , wherein said first and second times are separated by less than 1.5 seconds.

11. The display system of claim 1 , wherein the processing electronics are configured to alter said state of the head mounted display system based on a determination that the first and second eyes of the user transition from being in an opened state to being in an at least partially closed state at least three times, a first time, a second time, and a third time.

12. The display system of claim 11 , wherein said first and second times as well as said second and third times are separated by less than 3 seconds.

13. The display system of claim 11 , wherein said first and second times as well as said second and third times are separated by less than 2 seconds.

14. The display system of claim 11 , wherein said first and second times as well as said second and third times are separated by less than 1.5 seconds.

15. The display system of claim 1 , wherein the processing electronics are configured to display buttons, knobs, windows, menu items, or other regions toward which the user can direct their gaze to make a selection, the processing electronics being configured to identify the gaze direction of the first and second eyes and to use the gaze direction directed at the buttons, knobs, windows, menu items, or other regions and the determination that the first and second eyes of the user are at least partially closed in identifying a command or selection to alter the state of the head mounted display system so as to modify the virtual content presented to the user, the manner in which virtual content is presented to the user, a performance feature of the head mounted display system apparent to the user, or any combination thereof.

16. The display system of claim 1 , wherein the processing electronics are configured to display buttons, knobs, windows, menu items, or other regions toward which the user can direct their gaze to make a selection, the processing electronics being configured to identify the gaze direction and the vergence of said first and second eyes and to use the gaze direction and vergence directed at said buttons, knobs, windows, menu items, or other regions and said determination that the first and second eyes of the user are at least partially closed in identifying a command or selection to alter the state of the head mounted display system so as to modify the virtual content presented to the user, the manner in which virtual content is presented to the user, a performance feature of the head mounted display system apparent to the user, or any combination thereof.

17. The display system of claim 1 , wherein the processing electronics are configured to display buttons, knobs, windows, menu items, or other regions toward which the user can direct their gaze to make a selection, the processing electronics being configured to identify the vergence of said first and second eyes and to use the vergence directed at the buttons, knobs, windows, menu items, or other regions and the determination that the first and second eyes of the user are at least partially closed in identifying a command or selection to alter the state of the head mounted display system so as to modify the virtual content presented to the user, the manner in which virtual content is presented to the user, a performance feature of the head mounted display system apparent to the user, or any combination thereof.

18. The display system of 1 , wherein altering the manner in which virtual content is presented to the user comprises modifying the color of virtual content presented to the user, the color palette used to provide color to the virtual content, the position of the virtual content, or movement of the virtual content.

Assignments (6)
SECURITY INTEREST Recorded Oct 28, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073388/0027 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 23, 2023
From: MARTIN, HEATHER MICHELLE
To: MAGIC LEAP, INC.
Reel/Frame 065948/0261 →
EMPLOYMENT AGREEMENT Recorded Dec 23, 2023
From: DEOLIVEIRA, FLAVIO
To: MAGIC LEAP, INC.
Reel/Frame 066125/0699 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 4, 2023
From: ARROYO, PEDRO LUIS
To: MAGIC LEAP, INC.
Reel/Frame 065460/0446 →
EMPLOYMENT AGREEMENT Recorded Nov 4, 2023
From: O'BRIEN, KEVIN
To: MAGIC LEAP, INC.
Reel/Frame 065727/0648 →
SECURITY INTEREST Recorded May 24, 2022
From: MOLECULAR IMPRINTS, INC.; MENTOR ACQUISITION ONE, LLC; MAGIC LEAP, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 060338/0665 →
Continuity (3)
Provisional Application 62735369 · Sep 24, 2018
Provisional Application 62700775 · Jul 19, 2018
Related Publication 20210181514A1 · Jun 17, 2021