Identifying and displaying overlay markers for voice command user interface
View Patent ↗The present invention is a computer-implemented system and method that lets users access, navigate, and control a computer desktop and applications, windows, menus, icons, text entry fields, buttons, check-boxes, drop-down lists and HTML links and all other objects on the computer desktop, including those that are normally inaccessible to accessibility programs on a graphical user interface in a windowed computing environment.
1. A system to facilitate a graphical user interface in a computing environment, the system comprising a computing device configured to execute computer code in order to:
interact with a graphical user interface (GUI) in a window, the GUI comprising at least one user interface (UI) object accessible using a mouse;
receive a first command from a user to position a mouse cursor at a location in the window of the UI object in preparation for receiving a second command from the user to generate a marker representing the UI object, wherein the marker comprises a marker label;
receive the second command from the user;
display the marker label proximate the location of the UI object in the window; and
receive a voice command from the user that identifies the marker label, and in response to the voice command, generate an input representing the user interfacing with the UI object.
2. The system as recited in claim 1 , wherein the first command is received by the user physically moving a mouse to position the mouse cursor.
3. The system as recited in claim 1 , wherein the first command is received by a voice command from the user to position the mouse cursor.
4. The system as recited in claim 1 , wherein the second command is received by a voice command from the user.
5. The system as recited in claim 1 , wherein the second command is received by the user pressing a key on a keyboard.
6. The system as recited in claim 1 , wherein the marker label comprises a number.
7. The system as recited in claim 1 , wherein the computing device is further configured to execute computer code in order to:
generate an overlay comprising a plurality of markers, wherein each marker comprises a marker label; and
display the marker labels of the overlay in the window in response to receiving a third command from the user.
8. The system as recited in claim 1 , wherein the UI object comprises at least one of a menu command, an icon command, an icon tool, a check box, and a text input field.
9. The system as recited in claim 1 , wherein in response to the voice command the user interfacing with the UI object comprises emulating the user clicking a mouse cursor on the UI object.
10. A method of operating a system to facilitate a graphical user interface in a computing environment, the method comprising:
interacting with a graphical user interface (GUI) in a window, the GUI comprising at least one user interface (UI) object accessible using a mouse;
issuing a first command to position a mouse cursor at a location in the window of the UI object in preparation for issuing a second command to generate a marker representing the UI object, wherein the marker comprises a marker label;
issuing the second command; and
issuing a voice command that identifies the marker label in order to interface with the UI object.
11. The method as recited in claim 10 , wherein the first command is issued by physically moving a mouse to position the mouse cursor.
12. The method as recited in claim 10 , wherein the first command is issued by a voice command to position the mouse cursor.
13. The method as recited in claim 10 , wherein the second command is issued by a voice command.
14. The method as recited in claim 10 , wherein the second command is issued by pressing a key on a keyboard.
15. The method as recited in claim 10 , wherein the marker label comprises a number.
16. The method as recited in claim 10 , wherein the UI object comprises at least one of a menu command, an icon command, an icon tool, a check box, and a text input field.
17. The method as recited in claim 10 , wherein interfacing with the UI object comprises emulating a user clicking a mouse cursor on the UI object.
18. A non-transient computer readable medium comprising computer code to facilitate a graphical user interface in a computing environment, the computer code being configured to:
interact with a graphical user interface (GUI) in a window, the GUI comprising at least one user interface (UI) object accessible using a mouse;
receive a first command from a user to position a mouse cursor at a location in the window of the UI object in preparation for receiving a second command from the user to generate a marker representing the UI object, wherein the marker comprises a marker label;
receive the second command from the user;
display the marker label proximate the location of the UI object in the window; and
receive a voice command from the user that identifies the marker label, and in response to the voice command, generate an input representing the user interfacing with the UI object.
19. The non-transient computer readable medium as recited in claim 18 , wherein the first command is received by the user physically moving a mouse to position the mouse cursor.
20. The non-transient computer readable medium as recited in claim 18 , wherein the first command is received by a voice command from the user to position the mouse cursor.