Chatbot system for structured and unstructured data
Techniques for operating a chatbot system for enterprise-level conversational agents are disclosed. These techniques are performed by an application or cloud service executing on one or more computing devices. An enterprise system can deploy conversational agents onto user devices to run as chat interfaces for logging analytics question-answering. One example application or cloud service may be a multi-model chat mechanism configured to support these chat interfaces with backend functionality. In response to an incoming question, the chat mechanism first consolidates the question with any conversation history and then, classifies the user's question as either a question regarding unstructured document data, a question regarding structured log data, or a hybrid question. Based on the classification, the chat mechanism can generate a proper large language model (LLM) response.
1 . One or more non-transitory computer readable media comprising instructions which, when executed by one or more hardware processors, cause performance of operations comprising:
receiving, by an application or cloud service executing on one or more computing devices, user input provided via a chat interface running on a user device, the user input comprising a natural language question;
based on an application of a semantic parsing model to the natural language question, determining, by the application or cloud service, that the natural language question indicates at least one of a search of non-tabular data sources or a search of tabular data sources;
wherein determining that the natural language question indicates at least one of a search of non-tabular data sources or a search of tabular data sources is distinguished based on a semantic intent associated with the natural language question; and
generating, by the application or cloud service, computerized output for visualization on the user interface, the computerized output comprising at least one of:
a large language model (LLM) response to an LLM prompt that comprises retrieved data from the non-tabular data sources based on the natural language question, or
a textual representation of a structured data query that is generated from an application of a query language generator model to the natural language question and the tabular data sources.
2 . The one or more non-transitory computer readable media of claim 1 , the operations further comprising:
determining, by the application or cloud service, a consolidated question configured to maintain a conversation state of the user input.
3 . The one or more non-transitory computer readable media of claim 2 , wherein determining the consolidated question comprises generating the consolidated question as a combination of the natural language question and a conversation summary that results from an application of the LLM to the user input.
4 . The one or more non-transitory computer readable media of claim 1 , wherein the semantic parsing model comprises a transformer model for natural language classification.
5 . The one or more non-transitory computer readable media of claim 4 , wherein the semantic parsing model is fine-tuned for classifying the natural language question as a document question, a log question, or a hybrid question.
6 . The one or more non-transitory computer readable media of claim 4 , wherein the semantic parsing model comprises a DistilBERT™ model.
7 . The one or more non-transitory computer readable media of claim 1 , wherein the query language generating model is fine-tuned using field metadata of the tabular data sources.
8 . The one or more non-transitory computer readable media of claim 7 , wherein the query language generating model is configured to generate logging analytics query language queries.
9 . The one or more non-transitory computer readable media of claim 7 , wherein the query language generating model is fine-tuned using log field names.
10 . The one or more non-transitory computer readable media of claim 7 , wherein the query language generating model is fine-tuned using a custom query language.
11 . The one or more non-transitory computer readable media of claim 1 , wherein determining that the question indicates at least one of a search of non-tabular data sources or a search of tabular data sources further comprises determining that the question maps to at least one of keyword metadata used in the non-tabular data sources or field metadata used in the tabular data sources.
12 . The one or more non-transitory computer readable media of claim 1 , wherein generating the computerized output further comprises responsive to determining that the question indicates the search of non-tabular data sources, generating the LLM prompt from document fragments having one or more keywords from the natural language question.
13 . The one or more non-transitory computer readable media of claim 1 , wherein when the question indicates the search of tabular data sources, generating the structured query from field names that also are identified in the natural language question.
14 . The one or more non-transitory computer readable media of claim 1 , wherein the LLM prompt comprises retrieved data from the non-tabular data sources having a semantic similarity within a threshold of the natural language question.
15 . The one or more non-transitory computer readable media of claim 1 , wherein the LLM prompt comprises a retrieval augmented generation (RAG) pattern.
16 . The one or more non-transitory computer readable media of claim 15 , wherein the RAG pattern is retrieved from a data store comprising an entry, comprising document keyword metadata.
17 . The one or more non-transitory computer readable media of claim 16 , wherein at least one entry comprises log field metadata that maps to the document keyword metadata.
18 . The one or more non-transitory computer readable media of claim 16 , wherein the data store is populated with a plurality of document vectors from an embeddings model.
19 . A method comprising:
receiving, by an application or cloud service executing on one or more computing devices, user input provided via a chat interface running on a user device, the user input comprising a natural language question;
based on an application of a semantic parsing model to the natural language question, determining, by the application or cloud service, that the natural language question indicates at least one of a search of non-tabular data sources or a search of tabular data sources;
wherein determining that the natural language question indicates at least one of a search of non-tabular data sources or a search of tabular data sources is distinguished based on a semantic intent associated with the natural language question;
generating, by the application or cloud service, computerized output for visualization on the user interface, the computerized output comprising at least one of:
a large language model (LLM) response to an LLM prompt that comprises retrieved data from the non-tabular data sources, or
a textual representation of a structured data query that is generated from an application of a query language generator model to the natural language question and the tabular data sources; and
wherein the method is performed by at least one device including a hardware processor.
20 . A system comprising:
at least one device including a hardware processor; and
the system being configured to perform operations comprising:
receiving, by an application or cloud service executing on one or more computing devices, user input provided via a chat interface running on a user device the user input comprising a natural language question;
based on an application of a semantic parsing model to the natural language question, determining, by the application or cloud service executing on one or more computing devices, that the natural language question indicates at least one of a search of non-tabular data sources or a search of tabular data sources;
wherein determining that the natural language question indicates at least one of a search of non-tabular data sources or a search of tabular data sources is distinguished based on a semantic intent associated with the natural language question; and
generating, by the application or cloud service executing on one or more computing devices, computerized output for visualization on the user interface, the computerized output comprising at least one of:
a large language model (LLM) response to an LLM prompt that comprises retrieved data from the non-tabular data sources, or
a textual representation of a structured data query that is generated from an application of a query language generator model to the natural language question and the tabular data sources.