IP Library Granted Patent US 10,394,318
Granted Patent B2
US 10,394,318 · App. 15/286,222 · Granted Aug 27, 2019

Scene analysis for improved eye tracking

Inventor: Ezekiel Kruglick (Poway, CA)
Assignee: Empire Technology Development LLC
G06F3/013G02B27/017G02B27/0172G06K9/00604G06K9/00671G06K9/4642G06T7/74G02B27/0093G02B2027/014G02B2027/0138G02B2027/0178G02B2027/0187G06K2209/21G06T2207/30201
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,394,318
App. No.
15/286,222
Granted
Aug 27, 2019
Kind
B2
Abstract

Technologies related to scene analysis for improved eye tracking are generally described. In some examples, detected gaze targets may be derived from gaze direction information from an eye-facing sensor. Detected gaze target positions and/or motion may be improved by capturing and analyzing digital scene information from a scene visible by the eye. Digital scene information captured by a digital camera may be analyzed to identify potential gaze targets, such as stationary gaze targets, moving gaze targets, and/or accelerating gaze targets. Detected gaze targets may be modified to positions of selected gaze targets.

Claims (70)

1. An eye tracking method comprising:

receiving, by a computation device, gaze direction information for at least one eye from an eye-facing sensor;

detecting, by the computation device based on the gaze direction information, a gaze target corresponding to a scene visible to the at least one eye;

determining, by the computation device based on the gaze direction information position and motion information of the detected gaze target;

receiving, by the computation device from a digital camera, digital scene information, wherein the digital camera is oriented to capture the digital scene information from the scene visible to the at least one eye;

analyzing, by the computation device, the digital scene information to identify one or more potential gaze targets within the digital scene information, wherein the one or more potential gaze targets correspond to one or more objects in the scene visible to the at least one eye;

determining, by the computation device based on the analysis of the digital scene information, respective position and motion information of the one or more potential gaze targets within the digital scene information;

comparing, by the computation device, the position and motion information of the detected gaze target with the respective position and motion information of the one or more potential gaze targets to select a gaze target from among the one or more potential gaze targets;

modifying, by the computation device, the position and motion information of the detected gaze target to correspond to position and motion information of the selected gaze target, so as to correct the detected gaze target to correspond to the selected gaze target, wherein the selected gaze target is positioned as an object in the scene visible to the at least one eye and qualifies as a user input for the computation device, and wherein the detected gaze target does not qualify as a user input for the computation device; and

applying, by the computation device, the selected gaze target as the user input to perform at least one function associated with the computation device.

2. The eye tracking method of claim 1 , wherein the gaze direction information comprises angular coordinates.

3. The eye tracking method of claim 1 , wherein an estimated gaze direction corresponding to the selected gaze target is within an error boundary that surrounds the gaze direction information received from the eye-facing sensor.

4. The eye tracking method of claim 1 , wherein analyzing the digital scene information to identify the one or more potential gaze targets within the digital scene information comprises performing object recognition within the digital scene information to thereby determine the one or more potential gaze targets comprising recognized objects.

5. The eye tracking method of claim 1 , wherein analyzing the digital scene information to identify the one or more potential gaze targets within the digital scene information comprises identifying one or more sets of related graphical features within the digital scene information to determine the one or more potential gaze targets comprising sets of related graphical features.

6. The eye tracking method of claim 1 , wherein the one or more potential gaze targets comprise one or more of:

a stationary gaze target within the digital scene informational,

a moving gaze target within the digital scene information, or

an accelerating gaze target within the digital scene information.

7. The eye tracking method of claim 6 , further comprising:

modeling by the computation device, eye motions associated with following the moving gaze target or the accelerating gaze target in the scene visible to the at least one eye; and

comparing, by the computation device, modeled eye motions with eye motion detected at the eye-facing sensor to select the gaze target from among the one or more potential gaze targets.

8. The eye tracking method of claim 7 , wherein modeling the eye motions associated with following the moving gaze target or the accelerating gaze target in the scene visible to the at least one eye comprises applying an eye mechanical model simulator.

9. The eye tracking method of claim 1 , wherein selecting the gaze target from among the one or more potential gaze targets comprises dynamically assigning weights to a plurality of the one or more potential gaze targets, and using the dynamically assigned weights to select the gaze target.

10. The eye tracking method of claim 1 , further comprising determining, by the computation device based on the selected gaze target, augmented reality information for display.

11. The eye tracking method of claim 1 , wherein the digital camera has one or more of a default frame rate or a default resolution, and wherein the eye tracking method further comprises:

operating, by the computation device, the digital camera at one or more of a reduced frame rate, a reduced resolution, an enhanced frame rate, or an enhanced resolution, to capture the digital scene information, wherein the reduced frame rate is lower than the default frame rate, wherein the reduced resolution is lower than the default resolution, wherein the enhanced frame rate is higher than the default frame rate, and wherein the enhanced resolution is higher than the default resolution.

12. The eye tracking method of claim 1 , wherein the gaze direction information and the digital scene information, respectively, comprise real time gaze direction information and real time digital scene information, and wherein the eye tracking method further comprises:

modifying the selected gaze target in real time as the real time gaze direction information and real time digital scene information are received.

13. An eye tracking computation device, comprising:

a processor;

a memory; and

an eye tracker stored in the memory and executable by the processor, wherein the eye tracker is configured to perform operations that comprise:

receive gaze direction information for at least one eye from an eye-facing sensor;

detect, based on the gaze direction information, a gaze target corresponding to a scene visible to the at least one eye;

determine, based on the gaze direction information, position and motion information of the detected gaze target;

receive digital scene information from a digital camera, wherein the digital camera is oriented to capture the digital scene information from the scene visible to the at least one eye;

analyze the digital scene information to identify one or more potential gaze targets within the digital scene information, wherein the one or more potential gaze targets correspond to one or more objects in the scene visible to the at least one eye;

determine, based on the analysis of the digital scene information, respective position and motion information of the one or more potential gaze targets within the digital scene information;

compare the position and motion information of the detected gaze target with the respective position and motion information of the one or more potential gaze targets to select a gaze target from among the one or more potential gaze targets;

modify the position and motion information of the detected gaze target to correspond to position and motion information of the selected gaze target, so as to correct the detected gaze target to correspond to the selected gaze target, wherein the selected gaze target is positioned as an object in the scene visible to the at least one eye and qualifies as a user input for the eye tracking computation device, and

wherein the detected gaze target does not qualify as a user input for the eye tracking computation device; and

apply the selected gaze target as the user input to perform at least one function associated with the eye tracking computation device.

14. The eye tracking computation device of claim 13 ,

wherein the gaze direction information comprises angular coordinates.

15. The eye tracking computation device of claim 13 , wherein an estimated gaze direction corresponding to the selected gaze target is within an error boundary that surrounds the gaze direction information received from the eye-facing sensor.

16. The eye tracking computation device of claim 13 , wherein the one or more potential gaze targets comprise one or more of:

a stationary gaze target within the digital scene informational,

a moving gaze target within the digital scene in formation,

or an accelerating gaze target within the digital scene information.

17. The eye tracking computation device of claim 16 ,

wherein the operations further comprise:

model eye motions associated with following the moving gaze target or the accelerating gaze target in the scene visible to the at least one eye; and

compare modeled eye motions with eye motion detected at the eye-facing sensor to select the gaze target from among the one or more potential gaze targets.

18. The eye tracking computation device of claim 17 ,

wherein the operation to model the eye motions associated with following the moving gaze target or the accelerating gaze target in the scene visible to the at least one eye comprises at least one operation to apply an eye mechanical model simulator.

19. The eye tracking computation device of claim 13 , wherein the operations further comprise:

determine, based on the selected gaze target, augmented reality information for display.

20. The eye tracking computation device of claim 13 ,

wherein the gaze direction information and the digital scene information, respectively, comprise real time gaze direction information and real time digital scene information, and wherein the operations further comprise:

modify the selected gaze target in real time as the real time gaze direction information and real time digital scene information are received.

21. A non-transitory computer readable storage medium having computer executable instructions executable by a processor, wherein the instructions, in response to execution by the processor, cause the processor to perform operations that comprise:

obtain gaze direction information for at least one eye from an eye-facing sensor;

detect, based on the gaze direction information, a gaze target corresponding to a scene visible to the at least one eye;

determine, based on the gaze direction information, position and motion information of the detected gaze target;

obtain digital scene information from a digital camera, wherein the digital camera is oriented to capture the digital scene information from the scene visible to the at least one eye;

analyze the digital scene information to identify one or more potential gaze targets within the digital scene information, wherein the one or more potential gaze targets correspond to one or more objects in the scene visible to the at least one eye;

determine, based on the analysis of the digital scene information, respective position and motion information of the one or more potential gaze targets within the digital scene information;

compare the position and motion information of the detected gaze target with the respective position and motion information of the one or more potential gaze targets to select a gaze target from among the one or more potential gaze targets;

modify the position and motion information of the detected gaze target to correspond to position and motion information of the selected gaze target, so as to correct the detected gaze target to correspond to the selected gaze target, wherein the selected gaze target is positioned as an object in the scene visible to the at least one eye and qualifies as a user input for a computation device, and wherein the detected gaze target does not qualify as a user input for the computation device; and

apply the selected gaze target as the user input to perform at least one function associated with the computation device.

Assignments (3)
RELEASE OF SECURITY INTEREST IN PATENTS, RECORDED ON JANUARY 29, 2019, AT REEL/FRAME 048373/0217 Recorded Jun 22, 2026
From: CRESTLINE DIRECT FINANCE, L.P., AS COLLATERAL AGENT
To: EMPIRE TECHNOLOGY DEVELOPMENT LLC
Reel/Frame 075799/0053 →
RELEASE OF SECURITY INTEREST IN PATENTS, RECORDED ON JANUARY 29, 2019 AT REEL 048373 FRAME 0217 Recorded Sep 22, 2025
From: CRESTLINE DIRECT FINANCE, L.P., AS COLLATERAL AGENT
To: EMPIRE TECHNOLOGY DEVELOPMENT LLC
Reel/Frame 072936/0464 →
SECURITY INTEREST Recorded Jan 29, 2019
From: EMPIRE TECHNOLOGY DEVELOPMENT LLC
To: CRESTLINE DIRECT FINANCE, L.P.
Reel/Frame 048373/0217 →
Continuity (2)
Continuation 14458577 · Aug 13, 2014
Related Publication 20170024893A1 · Jan 26, 2017
Cited By (2)
US 12,474,771 US 12,608,078