Artificial intelligence application assistant
An Applicant Assistant System may receive, from a user device, a user input indicating a query or task associated with information displayed in a user interface. The device may determine one or more components associated with the user interface and access component metadata of the determined one or more components, the component metadata for each of the one or more components indicating one or more of a fact, a data link, or an action. A prompt comprising at least some of the component metadata, at least some context information, and an indication of one or more available response elements may be provided to a large language model (LLM). The LLM may response with one or more response elements that may be processed by the System, such as to cause updates to the user interface on the user device.
1. A method performed by an application assistant computing system, the method including:
receiving, from a user device, an input from a user indicating a query or task;
determining at least two components associated with a user interface displayed on the user device, each of the at least two components providing different user interface functionality;
accessing component metadata of the determined components;
accessing context information associated with the user or user device;
generating a natural language text prompt by combining at least some of the user input, text indicative of at least some of the component metadata, first text indicative of at least some of the context information, and second text indicative of one or more available response elements;
providing the prompt to a large language model (LLM);
receiving an output from the LLM indicating one or more response elements selected by the LLM, wherein a first of the one or more response elements indicates a function call;
generate a second prompt requesting executable code that interacts with an external service to perform the function call, the LLM prompt including one or more examples of outputs;
providing the second prompt to the LLM;
receiving a second output from the LLM including customized executable code that interacts with the external service in accordance with the query or task received from the user;
executing the customized executable code received from the LLM;
receiving, from the external service, additional response elements associated with the query or task received from the user;
generating, based on the one or more response elements and additional response elements, updated user interface code indicating one or more updates to the user interface displayed on the user device; and
initiating update of the user interface displayed on the user device according to the updated user interface code.
2. The method of claim 1 , wherein the component metadata for each of the at least two components indicating one or more of a fact, a data link, or an action.
3. The method of claim 1 , wherein a first response element of the one or more response elements includes a data link query of an external data source.
4. The method of claim 1 , wherein in response to determining that a size of the additional response elements is larger than a threshold size, generating and transmitting a processing prompt to the LLM requesting executable code configured to summarize or shorten the additional response elements.
5. The method of claim 1 , wherein the at least two components include one or more of: a navigation component, a notification component, a search component, a data item component, a visualization component, or a general component.
6. The method of claim 1 , wherein the context information indicates a user-selected component of the user interface.
7. The method of claim 6 , wherein the user-selected component is selected by the user selecting a chat button associated with the particular component.
8. The method of claim 6 , wherein the user-selected component is selected in response to the user hovering over the particular component.
9. The method of claim 1 , wherein at least some of the component metadata is included in metadata associated with the user interface.
10. The method of claim 9 , wherein the component metadata is transmitted from the user device to the application assistant computing system.
11. The method of claim 2 , wherein the fact includes one or more of information displayed in the user interface, information selected in the user interface, metadata associated with a visualization displayed in the user interface, a user identifier, a user role, or other user information.
12. The method of claim 2 , wherein one or more facts includes domain-specific knowledge usable by the LLM to better response to the user input.
13. The method of claim 2 , wherein the data link is usable to access a source of information displayed in the user interface.
14. The method of claim 2 , wherein the action indicates a user-initiated event including one or more of: initiating a search, initiating a sort, navigating to a different page, filling a form, submitting a form, moving data, or modifying data.
15. The method of claim 1 , wherein the updated user interface code includes a textual response to the query or task.
16. The method of claim 1 , wherein the prompt includes a contextual hint associated with a data access link, wherein the contextual hint indicates one or more of a required format, criteria to identify incorrect data, or examples of data that is acceptable.
17. The method of claim 1 , wherein the user input is initiated based on selection of a chat button or pressing a physical button on the user device.
18. A computing system comprising:
a hardware computer processor; and
a non-transitory computer readable medium having software instructions stored thereon, the software instructions executable by the hardware computer processor to cause the computing system to perform operations comprising:
receiving, from a user device, an input from a user indicating a query or task;
determining at least two components associated with a user interface displayed on the user device, each of the at least two components providing different user interface functionality;
accessing component metadata of the determined components;
accessing context information associated with the user or user device;
generating a natural language text prompt by combining at least some of the user input, text indicative of at least some of the component metadata, text indicative of at least some of the context information, and text indicative of one or more available response elements;
providing the prompt to a large language model (LLM);
receiving an output from the LLM indicating one or more response elements selected by the LLM, wherein a first of the one or more response elements indicates a function call;
generate a second prompt requesting executable code that interacts with an external service to perform the function call, the LLM prompt including one or more examples of outputs;
providing the second prompt to the LLM;
receiving a second output from the LLM including customized executable code that interacts with the external service in accordance with the query or task received from the user;
executing the customized executable code received from the LLM;
receiving, from the external service, additional response elements associated with the query or task received from the user;
generating, based on the one or more response elements and additional response elements, updated user interface code indicating one or more updates to the user interface displayed on the user device; and
initiating update of the user interface displayed on the user device according to the updated user interface code.