Controlling a computer using eyegaze and dwell
The systems and methods described herein assist persons with the use of computers based on eye gaze, and allow such persons to control such computing systems using various eye trackers. The systems and methods described herein use eye trackers to control cursor (or some other indicator) positioning on an operating system using the gaze location reported by the eye tracker. The systems and methods described herein utilize an interaction model that allows control of a computer using eye gaze and dwell. The data from eye trackers provides a gaze location on the screen. The systems and methods described herein control a graphical user interface that is part of an operating system relative to cursor positioning and associated actions such as Left-Click, Right-Click, Double-Click, and the like. The interaction model presents appropriate user interfaces to navigate the user through applications on the computing system.
1. A system for interacting with a user interface on a display, the system comprising:
a processor;
a memory communicatively coupled to the processor and including instructions, which when executed by the processor cause the system to perform operations comprising:
receiving data from a sensor, the sensor observes a gaze of a user;
detecting gaze coordinates of the user's gaze;
detecting the user's gaze within a predetermined threshold distance of the gaze coordinates for at least a predetermined time period;
displaying an indicator as an object within the user interface at the gaze coordinates once the user's gaze occurs for at least the predetermined time period at the gaze coordinates;
presenting an indirect indicator control window on the user interface, wherein the indirect indicator control window includes a transparent section to allow a user to see an area of the user interface at the gaze coordinates; and
wherein the indirect indicator control window includes at least one directional control object that, in response to being selected by the gaze of the user, moves the indirect indicator control window within the user interface.
2. The system of claim 1 , wherein the at least one directional control object includes multiple direction buttons that move the indicator within the user interface in a corresponding direction.
3. The system of claim 1 , wherein the indicator is a cursor.
4. The system of claim 1 , wherein the operations comprise:
detecting the presence of the sensor that observes the gaze of the user; and
outputting the user interface on the display, wherein the user interface includes at least one control object supporting gaze interaction with the user interface once the control object is activated by the user's gaze.
5. The system of claim 1 , wherein at least one of the processor and the memory includes an operating system that, which when executed by the processor cause the system to present the user interface on a display.
6. The system of claim 1 , wherein the indirect indicator control window includes control objects that, in response to being selected by the gaze of the user, perform actions relative to an item that is engaged by the indicator.
7. The system of claim 6 , wherein the actions performed relative to the item that is engaged by the indicator include at least one a left click function, a right click function, a double click function or a drag-and drop function.
8. A system for interacting with a user interface on a display, the system comprising:
a processor;
a memory communicatively coupled to the processor and including instructions, which when executed by the processor cause the system to perform the operations comprising:
receiving data from a sensor, the sensor to observe a gaze of a user;
detecting gaze coordinates of the user's gaze, the gaze coordinates being near a border of the display;
detecting the user's gaze within a predetermined threshold distance of the gaze coordinates for at least a predetermined time period;
displaying an indicator as an object within the user interface at the gaze coordinates once the user's gaze occurs for at least the predetermined time period at the gaze coordinates;
presenting an indirect indicator control window on the user interface that includes an at least partially transparent section to allow a user to see an area of the user interface at the gaze coordinates; and
wherein the partially transparent section of the indirect indicator control window shows a portion of the user interface near the border and a representation of at least a portion of the border of the display.
9. The system of claim 8 , wherein the portion of the border includes at least one of an edge of the border and a corner of the border.
10. The system of claim 8 , wherein the indicator is a cursor.
11. The system of claim 8 , wherein at least one of the processor and the memory includes an operating system that, which when executed by the processor cause the system to present the user interface on a display.
12. The system of claim 8 , wherein the operations comprise:
detecting the presence of the sensor that observes the gaze of the user; and
showing a user interface that includes at least one item near the border of the display that supports gaze interaction with the user interface once the item is activated by the user's gaze.
13. The system of claim 8 , wherein the indirect indicator control window is opaque, and wherein the indirect indicator control window reproduces contents of the user interface below the indirect indicator control window and renders an image of the user interface and a border of the user interface.
14. The system of claim 8 , wherein the indirect indicator control window includes control objects that, in response to being selected by the gaze of the user, move the indicator within the displayed area.
15. A system for interacting with a user interface on a display, the system comprising:
a processor;
a memory communicatively coupled to the processor and including instructions, which when executed by the processor cause the system to perform the operations comprising:
receiving data from a sensor, the sensor to observe a gaze of a user;
detecting gaze coordinates of the user's gaze;
detecting the user's gaze within a predetermined threshold distance of the gaze coordinates for at least a predetermined time period;
displaying an indicator as an object within the user interface at the gaze coordinates once the user's gaze occurs for at least the predetermined time period at the gaze coordinates;
presenting a control window on the user interface at the gaze coordinates; and
the control window includes control objects that; in response to being selected by the gaze of the user, perform actions that change the user interface.
16. The system of claim 15 , wherein the control window includes control objects that; in response to being selected by the gaze of the user, perform actions relative to an item that is engaged by the indicator on the user interface.
17. The system of claim 16 , wherein the actions performed relative to the item that is engaged by the indicator include at least one a left click function, a right click function, a double click function or a drag-and drop function.
18. The system of claim 15 , wherein the control window is an indirect indicator control window that includes a displayed area of the user interface, the indirect indicator control window including control objects that, in response to being selected by the gaze of the user, move the indirect indicator control window.
19. The system of claim 15 , wherein the indicator is a cursor and detecting eye control by the user at the gaze coordinates comprises detecting the user's gaze within a predetermined threshold distance of a set coordinates for at least a predetermined time period.
20. The system of claim 15 , wherein the operations comprise:
detecting the presence of the sensor that observes the gaze of the user; and
showing the user interface on the display that includes at least one item supporting gaze interaction with the user interface once the item is activated by the user's gaze.