IP Library Granted Patent US 11,226,681
Granted Patent B2
US 11,226,681 · App. 16/775,797 · Granted Jan 18, 2022

Gaze-based human interface controls

Inventor: Brent Vance Zucker (Roswell, GA)
Assignee: NCR Corporation
G06F3/013G06F3/017G06F3/0482G06F3/0484G06T7/20G06T2207/30201
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,226,681
App. No.
16/775,797
Granted
Jan 18, 2022
Kind
B2
Abstract

Eyes of an operator of a device are tracked with respect to an interface element of an interface presented on a display of the device. An interface operation that is directed to an interface element is determined based on a gaze of the eyes, a predefined movement of the eyes, and/or predefined movements of eyelids for the eye. The interface operation is processed on the interface element within the interface.

Claims (37)

1. A method, comprising:

tracking eyes of an individual with respect to an interface presented on a display of a device, wherein tracking further includes identifying window identifiers for each window presented within the display and identifying a topmost left corner coordinate pixel pair for each window presented within the display;

determining an interface operation based on the tracking;

processing the interface operation within the interface using the window identifiers and topmost left corner coordinate pixel pairs for the windows; and

processing the method as a logical Human-Interface Device (HID) without modification to the interface that is an existing interface.

2. The method of claim 1 , wherein tracking further includes determining an interface element that the eyes are focused on within the interface.

3. The method of claim 1 , wherein tracking further includes determining a predefined movement performed by one or more of the eyes.

4. The method of claim 1 , wherein tracking further includes determining an angle and a direction of the eyes from video frames captured by a camera interfaced to the device.

5. The method of claim 4 , wherein determining the angle and the direction further includes mapping the direction at the angle to an interface element presented within the interface on the display.

6. The method of claim 1 , determining further includes determining the interface option based on an elapsed time of a gaze of the eyes at a particular interface element presented within the interface on the display.

7. The method of claim 1 , wherein determining further includes determining the interface option based on a predefined movement of the eyes made in one or more directions or based on other predefined movements of one or more eyelids of the eyes.

8. The method of claim 1 , wherein processing the method further includes generating a HID-compliant format for the interface operation and processing the interface operation as a HID command issued from a HID device driver of the device.

9. The method of claim 1 , wherein processing the method further includes processing the logical HID for the interface being controlled by gazes of the eyes on interface elements presented within the interface on the display and controlled by movements of the eyes or movements of one or more eyelids of the eyes.

10. A method, comprising:

capturing video frames including eyes of an operator directed to a display that is presenting an interface;

mapping gazes or movements of the eyes to interface elements of the interface based on the video frames;

identifying interface operations associated with the interface elements based on the gazes or the movements, wherein identifying further includes identifying window identifiers for each window presented within the display and identifying a topmost left corner coordinate pixel pair for each window presented within the display;

causing the interface to process the interface operations on the interface elements using the window identifiers and the topmost left coordinate pixel pairs for the windows; and

processing the method as a logical Human-Interface Device (HID) without modification to the interface that is an existing interface.

11. The method of claim 10 , wherein capturing further includes obtaining display settings for the display and window settings for the windows a window presented on the display, the windows include that includes the interface.

12. The method of claim 11 , wherein obtaining further includes determining of distance of the eyes from the display and an angle of the eyes with respect to a display surface of the display based on the video frames.

13. The method of claim 12 , wherein mapping further includes determining a particular interface element associated with a particular gaze or a particular movement based on the display settings, the distance, and the angle.

14. The method of claim 10 , wherein identifying further includes identifying a first set of interface operations associated with a first set of interface elements of the interface but not presented within the interface and accessible through particular ones of the gazes or the movements.

15. The method of claim 11 , wherein identifying further includes identifying a second set of interface operations associated with a second set of interface elements of the interface that are presented within the interface and accessible through directed gazes or directed movements of the eyes focused on the second set of interface elements presented within the interface.

16. The method of claim 10 , wherein identifying further includes identifying a first set of interface operations based on particular ones of the movements that are associated with blinking of one or more eyelids for the eyes.

17. The method of claim 10 , wherein identifying further includes identifying a first set of interface operations based on an elapsed period of uninterrupted time during which particular ones of the gazes are maintained by the eyes of the operator on particular interface elements presented within the interface.

18. A system, comprising:

at least one camera configured to capture video frames comprise eyes of an operator of a device;

the device comprising:

a processor and non-transitory computer-readable storage medium having executable instructions representing an interface and an eye-based Human Interface Device (HID) controller for the interface;

the eye-based HID controller when executed by the processor from the non-transitory computer-readable storage medium causes the processor to perform processing comprising:

tracking gazes of the eyes, movements of the eyes, and second movements of eyelids of the eyes from the video frames captured by the at least one camera;

mapping the gazes, the movements, and the second movements to interface elements associated with the interface that is presented on a display of the device, wherein mapping further includes identifying window identifiers for each window presented within the display and identifying a topmost left corner coordinate pixel pair for each window presented within the display;

determining interface operations associated with the interface elements and recognized by the interface;

processing the interface operations with the interface to control the interface based on the gazes, the movements, and the second movements using the window identifiers and the topmost left corner coordinate pixel pairs for the windows; and

processing the interface and an eye-based Human Interface Device (HID) controller as a logical Human-Interface Device (HID) without modification to the interface that is an existing interface.

19. The system of claim 18 , wherein the device is: a mobile phone, a wearable processing device, a tablet, a Self-Service Terminal (SST), a Point-Of-Sale (POS) terminal, an Automated Teller Machine (ATM), a laptop, a desktop, a digital sign, a television, or a drive-thru terminal.

Assignments (3)
CHANGE OF NAME Recorded Dec 7, 2023
From: NCR CORPORATION
To: NCR VOYIX CORPORATION
Reel/Frame 065820/0704 →
SECURITY INTEREST Recorded Oct 25, 2023
From: NCR VOYIX CORPORATION
To: BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 065346/0168 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 29, 2020
From: ZUCKER, BRENT VANCE
To: NCR CORPORATION
Reel/Frame 051659/0281 →
Continuity (1)
Related Publication 20210232218A1 · Jul 29, 2021