Enabling natural language interactions with user interfaces for users of a software application

Inventors: Utku Yabas (Toronto, CA); Philipp Hubert (Toronto, CA); Karl Stahl (Palo Alto, CA)

Assignee: SOUNDHOUND AI IP, LLC

G10L15/22 G06F40/211 G06F40/284 G10L15/183 G10L15/26 G10L2015/223

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12670908
App. No.: 18/739,011
Granted: Jun 30, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

A user specifies a natural language command to a device. Software on the device generates contextual metadata about the user interface of the device, such as data about all visible elements of the user interface, and sends the contextual metadata along with the natural language command to a natural language understanding engine. The natural language understanding engine parses the natural language query using a stored grammar (e.g., a grammar provided by a maker of the device) and as a result of the parsing identifies information about the command (e.g., the user interface elements referenced by the command) and provides that information to the device. The device uses that provided information to respond to the command.

Claims (37)

1 . A computer-implemented method for identifying a user interface element referenced by a natural language command, the computer-implemented method comprising:

receiving, from a device, a natural language command and user interface metadata describing a set of user interface elements visible within a user interface of the device, the natural language command relating to selection of a user interface element of the set of user interface elements;

interpreting the natural language command using a grammar customized for the set of user interface elements, the customized grammar including rules for interpreting spatial positions of specific ones of the user interface elements and titles of specific ones of the user interface elements; and

determining the user interface element identified in the natural language command through interpretation of the natural language command using the customized grammar;

wherein the user interface metadata is specified in one of a key-value pair-based format or a schema-based markup language.

2 . The computer-implemented method of claim 1 , wherein the natural language command is a voice command, the method further comprising converting the voice command to text.

3 . The computer-implemented method of claim 1 , further comprising the step of selecting a grammar corresponding to the device.

4 . The computer-implemented method of claim 3 , wherein the grammar is a context-free grammar containing a single ordinal number non-terminal.

5 . The computer-implemented method of claim 3 , wherein selecting the grammar comprises identifying an organization associated with the device.

6 . The computer-implemented method of claim 3 , wherein selecting the grammar comprises identifying a functional type associated with the device.

7 . A computer-implemented method for interacting with a user interface of a device using a natural language command, the computer-implemented method comprising:

receiving a search request;

displaying, within a graphical user interface of the device, a set of search results corresponding to the search request;

receiving a natural language command from a user;

identifying a set of user interface elements visible within the graphical user interface of the device;

assembling user interface metadata describing the identified set of user interface elements;

interpreting the natural language command using a grammar customized for the set of user interface elements as described by the interface metadata, the customized grammar including rules for interpreting spatial positions of specific ones of the user interface elements;

determining a descriptor of a user interface element identified in the natural language command through interpretation of the natural language command using the customized grammar; and

using the descriptor to respond to the natural language command;

wherein the user interface metadata is specified in one of a key-value pair-based format or a schema-based markup language.

8 . The computer-implemented method of claim 7 , wherein the natural language command is a voice command.

9 . The computer-implemented method of claim 7 , wherein the descriptor is determined by a natural language understanding (NLU) engine.

10 . The computer-implemented method of claim 7 , wherein the user interface metadata include, for each identified user interface element, a caption of the element.

11 . A multi-tenant system comprising:

a computer processor; and

a non-transitory computer-readable storage medium storing instructions that when executed by the computer processor perform actions comprising:

identifying, based on tenant-specific information associated with the device, an organization or device type corresponding to the device;

selecting, based on the identified organization or device type, a grammar customized for the set of user interface elements;

determining the user interface element identified in the natural language command through interpretation of the natural language command using the customized grammar;

wherein the user interface metadata is specified in one of a key-value pair-based format or a schema-based markup language.

12 . The multi-tenant system of claim 11 , wherein the natural language command is a voice command, the method further comprising converting the voice command to text.

13 . The multi-tenant system of claim 11 , further comprising the steps of selecting a grammar corresponding to the device, the grammar having a slot, and parsing the natural language command according to the selected grammar, the slot being filled by text, the text being matched to the user interface element.

14 . The multi-tenant system of claim 13 , wherein the grammar is a context-free grammar containing two numerical non-terminals indicating a row and a column of a selected one of the visible user interface elements.

15 . The multi-tenant system of claim 13 , wherein selecting the grammar comprises identifying an organization associated with the device.

16 . The multi-tenant system of claim 13 , wherein selecting the grammar comprises identifying a functional type associated with the device.