IP Library › Granted Patent US 9,977,960
Granted Patent B2
US 9,977,960 · App. 15/276,448 · Granted May 22, 2018

Eye-tracking enabled wearable devices

Inventors: Simon Gustafsson (Danderyd, SE); Daniel Johansson Tornéus (Danderyd, SE); Jonas Andersson (Danderyd, SE); Johannes Kron (Danderyd, SE); Mårten Skogö (Danderyd, SE); Ralf Biedert (Danderyd, SE)
Assignee: Tobii AB
G06K9/00604G02B27/0093G02B27/017G02B27/0179G06F3/012G06F3/013G06F3/0304G06F3/0485G06F3/04815G06F3/04845G09G5/34A61B3/113G02B2027/014G02B2027/0138G02B2027/0187G06F2203/0381G09G2340/0464G09G2354/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,977,960
App. No.
15/276,448
Granted
May 22, 2018
Kind
B2
Abstract

A system for determining a gaze direction of a user is disclosed. The system may include a first illuminator, a first profile sensor, and at least one processor. The first illuminator may be configured to illuminate an eye of a user. The first profile sensor may be configured to detect light reflected by the eye of the user. The processor(s) may be configured to determine a gaze direction of the user based at least in part on light detected by the first profile sensor.

Claims (112)

1. A system for determining a gaze direction of a user, wherein the system comprises:

a first illuminator configured to illuminate an eye of a user;

a first profile sensor configured to detect light reflected by the eye of the user;

an area image sensor configured to detect light reflected by the eye of the user, wherein the area image sensor has a sensitive area having a total number of pixels characterized by a total number of rows and a total number of columns;

at least one processor configured to determine a gaze direction of the user based at least in part on light detected by the first profile sensor and based at least in part on light detected by the area image sensor;

wherein determining the gaze direction of the user based at least in part on light detected by the first profile sensor comprises:

determining a location of a glint on the eye caused by light reflected by the eye of the user, based at least in part on a higher magnitude light level detected at one or more pixels of the first profile sensor;

identifying a subset of a total number of pixels of the first profile sensor which include and surround the one or more pixels of the first profile sensor which detected the higher magnitude light level; and

analyzing only the subset of the total number of pixels of the first profile sensor during a subsequent determination of gaze direction; and

wherein determining the gaze direction of the user based at least in part on light detected by the area image sensor comprises:

determining magnitudes of light exposure at a subset of the total number of rows of the sensitive area during one or more illumination periods;

determining magnitudes of light exposure at a subset of the total number of columns of the sensitive area during one or more illumination periods;

identifying a subset of the total number of pixels of the sensitive area based on the magnitudes of light exposure at the subset of the total number of rows and the magnitudes of light exposure at the subset of the total number of columns; and

analyzing only the subset of the total number of pixels of the sensitive area during a subsequent illumination period.

2. The system for determining a gaze direction of a user of claim 1 , wherein the system further comprises:

a second profile sensor configured to detect light reflected by the eye of the user, wherein determining the gaze direction of the user is further based at least in part on light detected by the second profile sensor.

3. The system for determining a gaze direction of a user of claim 2 , wherein the at least one processor is further configured to:

determine a location of a cornea center of the eye based at least in part on light detected by the first profile sensor and light detected by the second profile sensor.

4. The system for determining a gaze direction of a user of claim 1 , wherein the first profile sensor is configured to:

detect light in two dimensions.

5. The system for determining a gaze direction of a user of claim 2 , wherein:

the first profile sensor is configured to detect light in one dimension; and

the second profile sensor is configured to detect light in one dimension.

6. The system for determining a gaze direction of a user of claim 5 , wherein:

the first profile sensor is orientated orthogonally to the second profile sensor.

7. The system for determining a gaze direction of a user of claim 5 , wherein the system further comprises:

a first cylindrical lens between the first profile sensor and the eye; and

a second cylindrical lens between the second profile sensor and the eye.

8. The system for determining a gaze direction of a user of claim 1 , wherein the system further comprises:

a second illuminator configured to illuminate the eye of the user.

9. The system for determining a gaze direction of a user of claim 8 , wherein the at least one processor is further configured to:

modulate activation of the first illuminator and the second illuminator such that only the first illuminator or the second illuminator is activated at a particular point in time.

10. The system for determining a gaze direction of a user of claim 1 , wherein:

the area image sensor operates at between about 0.5 and about 10.0 hertz.

11. A method for determining a gaze direction of a user, wherein the method comprises:

illuminating an eye of a user;

detecting, with an area image sensor, light reflected by the eye of the user, wherein the area image sensor has a sensitive area having a total number of pixels characterized by a total number of rows and a total number of columns;

determining, with at least one processor, a gaze direction of the user based at least in part on light detected by the area image sensor;

wherein determining the gaze direction of the user based at least in part on light detected by the area image sensor comprises:

determining a location of a glint on the eye caused by light reflected by the eye of the user, based at least in part on a higher magnitude light level detected at one or more pixels of the area image sensor;

determining magnitudes of light exposure at a subset of the total number of rows during one or more illumination periods;

determining magnitudes of light exposure at a subset of the total number of columns during one or more illumination periods;

based on the magnitudes of light exposure at the subset of the total number of rows and the magnitudes of light exposure at the subset of the total number of columns, identifying a subset of a total number of pixels of the area image sensor which include and surround the one or more pixels of the area image sensor which detected the higher magnitude light level; and

analyzing only the subset of the total number of pixels of the area image sensor during a subsequent determination of gaze direction during a subsequent illumination period.

12. A non-transitory machine readable medium having instructions stored thereon for determining a gaze direction of a user, wherein the instructions are executable by one or more processors to at least:

cause an illuminator to illuminate an eye of a user;

detect, with an area image sensor, light reflected by the eye of the user, wherein the area image sensor has a sensitive area having a total number of pixels characterized by a total number of rows and a total number of columns;

determine a gaze direction of the user based at least in part on light detected by the an area image sensor;

wherein determining the gaze direction of the user based at least in part on light detected by the an area image sensor comprises:

determining a location of a glint on the eye caused by light reflected by the eye of the user, based at least in part on a higher magnitude light level detected at one or more pixels of the an area image sensor;

determining magnitudes of light exposure at a subset of the total number of rows during one or more illumination periods;

determining magnitudes of light exposure at a subset of the total number of columns during one or more illumination periods;

based on the magnitudes of light exposure at the subset of the total number of rows and the magnitudes of light exposure at the subset of the total number of columns, identifying a subset of a total number of pixels of the an area image sensor which include and surround the one or more pixels of the profile sensor which detected the higher magnitude light level; and

analyzing only the subset of the total number of pixels of the an area image sensor during a subsequent determination of gaze direction during a subsequent illumination period.

13. A system for determining a gaze direction of a user, wherein the system comprises:

a first illuminator configured to illuminate an eye of a user;

an area image sensor configured to detect light reflected by the eye of the user, wherein the area image sensor has a sensitive area having a total number of pixels characterized by a total number of rows and a total number of columns; and

at least one processor configured to determine a gaze direction of the user based at least in part on light detected by the area image sensor;

wherein determining the gaze direction of the user based at least in part on light detected by the area image sensor comprises:

determining a location of a glint on the eye caused by light reflected by the eye of the user, based at least in part on a higher magnitude light level detected at one or more pixels of the area image sensor;

determining magnitudes of light exposure at a subset of the total number of rows during one or more illumination periods;

determining magnitudes of light exposure at a subset of the total number of columns during one or more illumination periods;

based on the magnitudes of light exposure at the subset of the total number of rows and the magnitudes of light exposure at the subset of the total number of columns, identifying a subset of a total number of pixels of the area image sensor which include and surround the one or more pixels of the area image sensor which detected the higher magnitude light level; and

analyzing only the subset of the total number of pixels of the area image sensor during a subsequent determination of gaze direction during a subsequent illumination period.

14. The system of claim 13 , wherein the system further comprises:

a Fresnel lens in front of a display device;

a hot mirror on at least one side of the Fresnel lens or the display device;

and wherein:

the at least one processor is further configured to adjust the trajectory of virtual missiles which are caused to be virtually propelled by the user in virtual space; and

the first illuminator and image sensor are removable from the system so that they may be used in other devices.

15. A system for determining a gaze direction of a user, wherein the system comprises:

a first illuminator configured to illuminate an eye of a user;

a first profile sensor configured to detect light reflected by the eye of the user;

an area image sensor configured to detect light reflected by the eye of the user, wherein the area image sensor has a sensitive area having a total number of pixels characterized by a total number of rows and a total number of columns;

at least one processor configured to determine a gaze direction of the user based at least in part on light detected by the first profile sensor and based at least in part on light detected by the area image sensor;

wherein determining the gaze direction of the user based at least in part on light detected by the first profile sensor comprises:

determining a location of a glint on the eye caused by light reflected by the eye of the user, based at least in part on a higher magnitude light level detected at one or more pixels of the first profile sensor;

identifying a subset of a total number of pixels of the first profile sensor which include and surround the one or more pixels of the first profile sensor which detected the higher magnitude light level; and

analyzing only the subset of the total number of pixels of the first profile sensor during a subsequent determination of gaze direction; and

wherein determining the gaze direction of the user based at least in part on light detected by the area image sensor comprises:

determining magnitudes of light exposure at a subset of pixels in each of the total number of rows of the sensitive area during one or more illumination periods;

determining magnitudes of light exposure at a subset of pixels in each of the total number of columns of the sensitive area during one or more illumination periods;

identifying a subset of the total number of pixels of the sensitive area based on the magnitudes of light exposure at the subset of pixels in each of the total number of rows and the magnitudes of light exposure at the subset of pixels in each of the total number of columns; and

analyzing only the subset of the total number of pixels of the of the sensitive area during a subsequent illumination period.

16. A system for determining a gaze direction of a user, wherein the system comprises:

a first illuminator configured to illuminate an eye of a user, the first illuminator comprising a near-infrared illuminator;

a first profile sensor configured to detect light reflected by the eye of the user;

at least one processor configured to determine a gaze direction of the user based at least in part on light detected by the first profile sensor;

wherein at least the first illuminator and the first profile sensor are disposed within a wearable device comprising a display device and a cold mirror on a back side of the display device, wherein the cold mirror is configured to reflect visible light toward a front side of the display device, and allow near-infrared to pass through the cold mirror, and wherein the first profile sensor is disposed on the opposite side of the cold mirror from the display device; and

wherein determining the gaze direction of the user based at least in part on light detected by the first profile sensor comprises:

determining a location of a glint on the eye caused by light reflected by the eye of the user, based at least in part on a higher magnitude light level detected at one or more pixels of the first profile sensor;

identifying a subset of a total number of pixels of the first profile sensor which include and surround the one or more pixels of the first profile sensor which detected the higher magnitude light level; and

analyzing only the subset of the total number of pixels of the first profile sensor during a subsequent determination of gaze direction.

17. The system for determining a gaze direction of a user of claim 16 , wherein:

the wearable device further comprises a lens in front of the display device; and

the at least one processor is further configured to compensate for the presence of the lens before or during determining the gaze direction of the user.

18. A system for determining a gaze direction of a user, wherein the system comprises:

a first illuminator configured to illuminate an eye of a user;

an area image sensor configured to detect light reflected by the eye of the user, wherein the area image sensor has a sensitive area having a total number of pixels characterized by a total number of rows and a total number of columns; and

at least one processor configured to determine a gaze direction of the user based at least in part on light detected by the area image sensor;

wherein determining the gaze direction of the user based at least in part on light detected by the area image sensor comprises:

determining a location of a glint on the eye caused by light reflected by the eye of the user, based at least in part on a higher magnitude light level detected at one or more pixels of the area image sensor;

determining magnitudes of light exposure at a subset of pixels in each of the total number of rows during one or more illumination periods;

determining magnitudes of light exposure at a subset of pixels in each of the total number of columns during one or more illumination periods;

based on the magnitudes of light exposure at the subset of pixels in each of the total number of rows and the magnitudes of light exposure at the subset of pixels in each of the total number of columns, identifying a subset of a total number of pixels of the area image sensor which include and surround the one or more pixels of the area image sensor which detected the higher magnitude light level; and

analyzing only the subset of the total number of pixels of the area image sensor during a subsequent determination of gaze direction during a subsequent illumination period.

19. The system of claim 18 , wherein the system further comprises:

a Fresnel lens in front of a display device;

a hot mirror on at least one side of the Fresnel lens or the display device;

and wherein:

the at least one processor is further configured to adjust the trajectory of virtual missiles which are caused to be virtually propelled by the user in virtual space; and

the first illuminator and image sensor are removable from the system so that they may be used in other devices.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 28, 2016
From: GUSTAFSSON, SIMON; JOHANSSON TORNÉUS, DANIEL; ANDERSSON, JONAS; KRON, JOHANNES; SKOGÖ, MÅRTEN; BIEDERT, RALF
To: TOBII AB
Reel/Frame 040784/0653 →
Continuity (3)
Provisional Application 62329719 · Apr 29, 2016
Provisional Application 62232268 · Sep 24, 2015
Related Publication 20170091549A1 · Mar 30, 2017