Enabling natural language interactions with user interfaces for users of a software application
A user specifies a natural language command to a device. Software on the device generates contextual metadata about the user interface of the device, such as data about all visible elements of the user interface, and sends the contextual metadata along with the natural language command to a natural language understanding engine. The natural language understanding engine parses the natural language query using a stored grammar (e.g., a grammar provided by a maker of the device) and as a result of the parsing identifies information about the command (e.g., the user interface elements referenced by the command) and provides that information to the device. The device uses that provided information to respond to the command.
1. A computer-implemented method for identifying a user interface element referenced by a natural language command, the computer-implemented method comprising:
receiving, from a remote device over a network, a natural language command and user interface metadata describing a set of user interface elements visible within a user interface of the device;
selecting a grammar corresponding to the device, the grammar being a context-free grammar having a slot and containing two numerical non-terminals indicating a row and a column of a selected one of the visible user interface elements;
parsing the natural language command according to the selected grammar, the slot being filled by text;
matching the text with a user interface element in the user interface metadata; and
providing a descriptor of the matched user interface element to the device.
2. The computer-implemented method of claim 1 , wherein the natural language command is a voice command, the method further comprising converting the voice command to text.
3. The computer-implemented method of claim 1 , wherein the user interface metadata is specified in one of a key-value pair-based format or a schema-based markup language.
4. The computer-implemented method of claim 1 , wherein the grammar is a context-free grammar containing a single ordinal number non-terminal.
5. The computer-implemented method of claim 1 , wherein selecting the grammar comprises identifying an organization associated with the device.
6. The computer-implemented method of claim 1 , wherein selecting the grammar comprises identifying a functional type of the device.
7. A computer-implemented method for interacting with a user interface of a device using a natural language command, the computer-implemented method comprising:
receiving a search request;
displaying, within a graphical user interface of the device, a set of search results corresponding to the search request;
receiving a natural language command from a user;
identifying a set of user interface elements visible within the graphical user interface of the device;
assembling user interface metadata describing the identified set of user interface elements;
providing the language command and the user interface metadata to a natural language understanding (NLU) engine;
receiving, from the NLU engine, a descriptor of a user interface element that is within the set of user interface elements and that is referenced by the natural language command, wherein the NLU engine uses a grammar corresponding to the device to determine the descriptor of the user interface element, the grammar being a context-free grammar having a slot and containing two numerical non-terminals indicating a row and a column of a selected one of the visible user interface elements; and
using the descriptor to respond to the natural language command.
8. The computer-implemented method of claim 7 , wherein the natural language command is a voice command.
9. The computer-implemented method of claim 7 , wherein the user interface metadata is specified in one of a key-value pair-based format or a schema-based markup language.
10. The computer-implemented method of claim 7 , wherein the user interface metadata include, for each identified user interface element, a caption of the element.
11. A multi-tenant system comprising:
a computer processor; and
a non-transitory computer-readable storage medium storing instructions that when executed by the computer processor perform actions comprising:
receiving, from a device, a natural language command and user interface metadata describing a set of user interface elements visible within a user interface of the device;
selecting a grammar corresponding to the device, the grammar being a context-free grammar and having a slot and containing two numerical non-terminals indicating a row and a column of a selected one of the visible user interface elements;
parsing the natural language command according to the selected grammar, the slot being filled by text;
matching the text with a user interface element in the user interface metadata; and
providing a descriptor of the matched user interface element to the device.
12. The multi-tenant system of claim 11 , wherein the natural language command is a voice command, the instructions further comprising converting the voice command to text.
13. The multi-tenant system of claim 11 , wherein the user interface metadata is specified in one of a key-value pair-based format or a schema-based markup language.
14. The multi-tenant system of claim 11 , wherein the grammar is a context-free grammar containing a single ordinal number non-terminal.
15. The multi-tenant system of claim 11 , wherein selecting the grammar comprises identifying an organization associated with the device.
16. The multi-tenant system of claim 11 , wherein selecting the grammar comprises identifying a functional type associated with the device.
17. The computer-implemented method of claim 7 , further comprising:
analyzing a gaze of the user to determine whether the user is looking at the graphical user interface of the device; and
performing the identifying, assembling, providing, receiving, and using responsive to determining that the user is looking at the graphical user interface of the device.