IP Library Granted Patent US 10,757,328
Granted Patent B2
US 10,757,328 · App. 15/390,410 · Granted Aug 25, 2020

Eye tracking using video information and electrooculography information

Inventors: Vaibhav Thukral (Kirkland, WA); Chris Aholt (Newcastle, WA); Christopher Maurice Mei (Redmond, WA); Bill Chau (Sammamish, WA); Nguyen Bach (Kirkland, WA); Lev Cherkashin (Redmond, WA); Jaeyoun Kim (Fremont, CA)
Assignee: MICROSOFT TECHNOLOGY LICENSING, LLC
H04N5/23241A61B3/0025A61B3/113A61B3/145A61B5/0496G06F1/163G06F1/1686G06F1/3203G06F1/325G06F3/011G06F3/013G06F3/015G06F3/0304G06K9/00604G06T7/0014G06T7/248G06T7/292G06T7/74G06T2207/30041
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,757,328
App. No.
15/390,410
Granted
Aug 25, 2020
Kind
B2
Abstract

Disclosed are an apparatus and a method of low-latency, low-power eye tracking. In some embodiments, the eye tracking method operates a first sensor having a first level of power consumption that tracks positions of an eye of a user. In response to detection that the eye does not change position for a time period, the method stops operation of the first sensor and instead operates a second sensor that detects a change of position of the eye. The second sensor has a level of power consumption lower than the level of power consumption of the first sensor. Once the eye position changes, the second sensor resumes operation.

Claims (49)

1. A method of eye tracking, comprising:

operating a video oculography (VOG) sensor to track a position of an eye of a user;

stopping operation of the VOG sensor in response to a first detection, based on VOG signals from the VOG sensor, that the eye does not change position for a time period;

operating an electrooculography (EOG) sensor to track the position of the eye responsive to stopping the operation of the VOG sensor, wherein the EOG sensor has a lower power consumption than the VOG sensor; and

resuming operation of the VOG sensor in response to a second detection, based on EOG signals from the EOG sensor, of a change of the position of the eye.

2. The method of claim 1 , wherein the VOG sensor has a first signal processing latency that is higher than a second signal processing latency of the EOG sensor.

3. The method of claim 1 , wherein the VOG sensor has a first eye-tracking accuracy that is greater than a second eye-tracking accuracy of the EOG sensor.

4. The method of claim 1 , wherein the VOG sensor comprises a camera that captures images of the eye of the user.

5. The method of claim 1 , wherein the EOG sensor comprises a low-power camera.

6. The method of claim 1 , further comprising:

estimating two eye positions based on the EOG signals of the EOG sensor at two frames and calculating a difference between the two eye positions; and

recognizing the change of the position of the eye when the difference between the two eye positions exceeds a threshold.

7. The method of claim 1 , further comprising:

continuing operation of the EOG sensor when the operation of the EOG sensor resumes.

8. An electronic device for eye tracking, comprising:

a viedo oculography (VOG) sensor to track a position of an eye of a user;

an electrooculography (EOG) sensor to track the position of the eye of the user, wherein the EOG sensor has a lower power consumption than the VOG sensor; and

a processor configured to execute a process including:

detecting, based on VOG signals from the VOG sensor, that the eye does not change position for a time period;

stopping operation of the VOG senror in response to detecting that the eye does not change position for the time period;

triggering the EOG sensor to track the position of the eye responsive to stopping the operation of the VOG sensor; and

resuming operation of the VOG sensor in response to a detection, based on EOG signals of the EOG sensor, of a change of the position of the eye.

9. The electronic device of claim 8 , further comprising:

another VOG sensor that tracks positions of another eye of the user; and

another EOG sensor that tracks the positions of the another eye of the user.

10. The electronic device of claim 8 , further comprising:

a light source that generates light that is reflected by the eye and captured by the VOG sensor or the EOG sensor.

11. The electronic device of claim 8 , further comprising:

a control circuit that controls an operation of the VOG sensor based on instructions from the processor.

12. The electronic device of claim 8 , wherein the processor is further configured to:

estimate two eye positions based on the EOG signals of the EOG sensor at two frames;

calculate a difference between the two eye positions; and

recognize the change of the position of the eye when the difference between the two eye positions exceeds a threshold.

13. The electronic device of claim 8 , wherein the VOG sensor has a first signal processing latency that is higher than a second signal processing latency of the EOG sensor, or the VOG sensor has a first eye-tracking accuracy that is better than a second eye-tracking accuracy of the EOG sensor.

14. The electronic device of claim 8 , where the VOG sensor is a camera, and the EOG sensor comprises a low-power camera.

15. A method of eye tracking, comprising:

operating a video oculography (VOG) sensor to track positions of an eye of a user at a first frame rate;

determining the positions of the eye at keyframes based on VOG signals collected from the VOG sensor at the first frame rate, wherein the VOG sensor stops operating based on the eye remaining in a fixed position for a time period;

operating am electrooculography (EOG) sensor to track the positions of the eye of the user at a second frame rate upon stopping an operation of the VOG sensor, wherein the EOG sensor has a lower power consumption than the VOG sensor, wherein the first frame rate is lower than the second frame rate;

determining the positions of the eye at regular frames based on EOG signals collected from the EOG sensor at the second frame rate;

determining a fixation point at a user interface based on a current position from the fixed position of the eye being tracked;

conducting a foveated rendering of the user interface based on the fixation point; and

resuming operation of the VOG sensor in response to a detection, based on the EOG signals from the EOG sensor, of a change of a position of the eye.

16. The method of claim 15 , further comprising:

generating a bias between an eye position determined based on the VOG signals collected from the VOG sensor at a keyframe and another eye position determined based on the EOG signals collected from the EOG sensor at a regular frame; and

calibrating multiple eye positions determined based on the EOG signals from the EOG sensor by subtracting the bias from the multiple eye positions.

17. The method of claim 15 , wherein the VOG sensor has a first signal processing latency that is higher than a second signal processing latency of the EOG sensor.

18. The method of claim 15 , wherein the VOG sensor has a first eye-tracking accuracy that is better than a second eye-tracking accuracy of the EOG sensor.

19. The method of claim 15 , wherein the EOG sensor comprises a low-power camera sensor.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 18, 2017
From: THUKRAL, VAIBHAV; AHOLT, CHRIS; MEI, CHRISTOPHER MAURICE; CHAU, BILL; BACH, NGUYEN; CHERKASHIN, LEV; KIM, JAEYOUN
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 043616/0752 →
Continuity (1)
Related Publication 20180184002A1 · Jun 28, 2018
Cited By (3)
US 12,277,268 US 12,282,598 US 12,468,387