Enhanced search for information discovery
A hybrid architecture that combines the efficiency of small language models with the accuracy of large language models for enhanced selection of requested functionality and identification of data visualizations using an enterprise system is described. For instance, a widget selection method employs a hybrid architecture using both small and large language model to efficiently search through possible widgets by using an LLM to vectorize the titles widgets and information associated with those widgets and a small language model to use a natural language model to search the vectorized widgets.
1 . A method for generating a payload response for a requested functionality and a data visualization for the requested functionality in a system environment, the method comprising:
receiving, at a network system from a client device, a request comprising a natural language request for the data visualization of the requested functionality;
selecting one or more datastores from a plurality of datastores based on an analysis of the natural language request to determine a data domain associated with the requested functionality, wherein each datastore is associated with a different data domain and each of the data stores comprises a data corpus including structured data describing at least one of transactions, metrics, or time-series data for the corresponding data domain;
determining, at the network system, a set of selected data visualizations from the selected one or more datastores, each of the selected data visualizations configured for displaying information about the requested functionality, the determination comprising:
generating a query vector representing the natural language request using a small language model;
comparing the query vector to each of a plurality of stored enhanced vectors from the selected one or more datastores, each stored enhanced vector corresponding to a data visualization of a plurality of data visualizations, and the plurality of stored enhanced vectors generated by applying a large language model to at least a title for each of the plurality of data visualizations to extract semantic content, context, and intent of the title; and
selecting one or more of the plurality of data visualizations as the set of selected data visualizations based on the comparison; and
providing, from the network system to the client device, the set of selected data visualizations to the client device for visualizing data from the data corpus.
2 . The method of claim 1 , wherein selecting the set of selected data visualization based on the comparison comprises calculating a similarity between the query vector and each of the plurality of stored enhanced vectors corresponding to data visualizations.
3 . The method of claim 2 , wherein the set of selected data visualizations comprises a number of data visualizations having a calculated similarity above a threshold.
4 . The method of claim 2 , wherein the set of selected data visualizations comprises a predetermined number of data visualizations.
5 . The method of claim 1 , further comprising:
applying a model to each of the plurality of data visualizations to determine additional information describing the plurality of data visualizations; and
generating a stored enhanced vector representing the additional information for each of the plurality of data visualizations.
6 . The method of claim 5 , wherein the model is a large language model and the additional information is information associated with content, context, and intent of a title of the data visualization.
7 . The method of claim 5 , wherein the additional information comprises one or more of people, events, metrics, or entities associated with a title of the data visualization.
8 . The method of claim 1 , further comprising:
requesting, from one or more additional client devices, data for generating a data corpus, the data corpus comprising information which the plurality of data visualizations are configured to visualize.
9 . The method of claim 1 , further comprising:
receiving, from one or more additional client devices, one or more data visualizations for the plurality of data visualizations.
10 . The method of claim 1 , further comprising:
selecting one or more datastores from a plurality of datastores, the selection based on a comparison of visualizations stored in each of the plurality of datastores to the requested functionality of the natural language request; and
selecting the plurality of data visualizations from the selected one or more datastores.
11 . A non-transitory computer-readable storage medium comprising computer program instructions for generating a payload response for a requested functionality and a data visualization for the requested functionality in a system environment, the computer program instructions, when executed by one or more processors, causing the one or more processors to:
receive, at a network system from a client device, a request comprising a natural language request for the data visualization of the requested functionality;
select one or more datastores from a plurality of datastores based on an analysis of the natural language request to determine a data domain associated with the requested functionality, wherein each datastore is associated with a different data domain and each of the data stores comprises a data corpus including structured data describing at least one of transactions, metrics, or time-series data for the corresponding data domain;
determine, at the network system, a set of selected data visualizations from the selected one or more datastores, each of the selected data visualizations configured for displaying information about the requested functionality, the determination comprising:
generating a query vector representing the natural language request using a small language model;
comparing the query vector to each of a plurality of stored enhanced vectors from the selected one or more datastores, each stored enhanced vector corresponding to a data visualization of a plurality of data visualizations, and the plurality of stored enhanced vectors generated by applying a large language model to at least a title for each of the plurality of data visualizations to extract semantic content, context, and intent of the title; and
selecting one or more of the plurality of data visualizations as the set of selected data visualizations based on the comparison; and
provide, from the network system to the client device, the set of selected data visualizations to the client device for visualizing data from the data corpus.
12 . The non-transitory computer-readable storage medium of claim 11 , wherein selecting the set of selected data visualization based on the comparison causes the one or more processors to:
calculate a similarity between the query vector and each of the plurality of stored enhanced vectors corresponding to data visualizations.
13 . The non-transitory computer-readable storage medium of claim 12 , wherein the set of selected data visualizations comprises a number of data visualizations having a calculated similarity above a threshold.
14 . The non-transitory computer-readable storage medium of claim 12 , wherein the set of selected data visualizations comprises a predetermined number of data visualizations.
15 . The non-transitory computer-readable storage medium of claim 11 , wherein the computer program instructions, when executed by the one or more processors, cause the one or more processors to:
apply a model to each of the plurality of data visualizations to determine additional information describing the plurality of data visualizations; and
generate a stored enhanced vector representing the additional information for each of the plurality of data visualizations.
16 . The non-transitory computer-readable storage medium of claim 15 , wherein the model is a large language model and the additional information is information associated with content, context, and intent of a title of the data visualization.
17 . The non-transitory computer-readable storage medium of claim 15 , wherein the additional information comprises one or more of people, events, metrics, or entities associated with a title of the data visualization.
18 . The non-transitory computer-readable storage medium of claim 11 , wherein the computer program instructions, when executed by the one or more processors, cause the one or more processors to:
request, from one or more additional client devices, data for generating a data corpus, the data corpus comprising information which the plurality of data visualizations are configured to visualize.
19 . The non-transitory computer-readable storage medium of claim 11 , wherein the computer program instructions, when executed by the one or more processors, cause the one or more processors to:
receive, from one or more additional client devices, one or more data visualizations for the plurality of data visualizations.
20 . The non-transitory computer-readable storage medium of claim 11 , wherein the computer program instructions, when executed by the one or more processors, cause the one or more processors to:
select one or more datastores from a plurality of datastores, the selection based on a comparison of visualizations stored in each of the plurality of datastores to the functionality requested in the natural language request; and
select the set of visualization from the selected one or more datastores.