Using sample question embeddings to choose between an LLM interfacing model and a non-LLM interfacing model
Methods, systems, and computer program products for managing interactions between a content management system (CMS) and a large language model (LLM) system. The semantics of user questions can be considered before prompting an LLM, or alternatively, before querying datasets that are local to the CMS. Given a user question to be answered, the embedding of the user question can be matched against preconfigured sample question embeddings to determine a best match. A prompt corresponding to the determined best match is then configured based on identification of the class or classes that correspond to the matched question. Prompts for provision to LLMs can be synthesized based on a particular user's identity and/or based on the particular user's historical collaboration activities over objects of the CMS. The LLM can be hosted by a third-party provider. Alternatively all or portions of a large language model system can be hosted within the CMS.
1 . A method for choosing between a large language model agent (LLM agent) or a non-LLM agent, the method comprising:
configuring a content management system (CMS) to expose content objects stored in a shared content repository to a plurality of content management system users;
receiving a user question from a content management system user of the plurality of content management system users;
responsive to receipt of the user question, selecting at least one computing agent from a plurality of candidate computing agents, wherein the plurality of candidate computing agents comprise at least a LLM agent and a non-LLM agent, and the at least one computing agent is selected at least based on results of processing the user question to compare the user question to data stored in a database that represents example questions and is separate from the shared content repository; and
processing the user question by:
when the LLM agent is selected, forming an answer to the user question at least by formulating an LLM prompt using the LLM agent, and sending the LLM prompt to an LLM system, and
when the non-LLM agent is selected, deriving an answer to the user question based on at least characteristics of CMS data associated with at least a subset of the content objects stored in the shared content repository.
2 . The method of claim 1 , further comprising determining a domain-specific query language.
3 . The method of claim 2 , further comprising executing a domain-specific query to produce local query results, the domain-specific query based at least in part on the domain-specific query language.
4 . The method of claim 3 , further comprising reformatting the local query results to render the local query results into natural language.
5 . The method of claim 3 , further comprising identifying a preconfigured template.
6 . The method of claim 5 , wherein the identifying of the preconfigured template comprises comparing at least one best match value against a minimum match threshold.
7 . The method of claim 6 , wherein, when the minimum match threshold is not met, then a generic prompt is used.
8 . The method of claim 1 , wherein the deriving an answer to the user question is performed without formulating an LLM prompt.
9 . A non-transitory computer readable medium having stored thereon a sequence of instructions which, when stored in memory and executed by one or more processors causes the one or more processors to perform a set of acts for choosing between a large language model agent (LLM agent) or a non-LLM agent, the set of acts comprising:
configuring a content management system (CMS) to expose content objects stored in a shared content repository to a plurality of content management system users;
receiving a user question from a content management system user of the plurality of content management system users;
responsive to receipt of the user question, selecting at least one computing agent from a plurality of candidate computing agents, wherein the plurality of candidate computing agents comprise at least a LLM agent and a non-LLM agent, and the at least one computing agent is selected at least based on results of processing the user question to compare the user question to data stored in a database that represents example questions and is separate from the shared content repository; and
processing the user question by:
when the LLM agent is selected, forming an answer to the user question at least by formulating an LLM prompt using the LLM agent, and sending the LLM prompt to an LLM system, and
when the non-LLM agent is selected, deriving an answer to the user question based on at least characteristics of CMS data associated with at least a subset of the content objects stored in the shared content repository.
10 . The non-transitory computer readable medium of claim 9 , further comprising instructions which, when stored in memory and executed by the one or more processors causes the one or more processors to perform acts of determining a domain-specific query language.
11 . The non-transitory computer readable medium of claim 10 , further comprising instructions which, when stored in memory and executed by the one or more processors causes the one or more processors to perform acts of executing a domain-specific query to produce local query results, the domain-specific query based at least in part on the domain-specific query language.
12 . The non-transitory computer readable medium of claim 11 , further comprising instructions which, when stored in memory and executed by the one or more processors causes the one or more processors to perform acts of reformatting the local query results to render the local query results into natural language.
13 . The non-transitory computer readable medium of claim 11 , further comprising instructions which, when stored in memory and executed by the one or more processors causes the one or more processors to perform acts of identifying a preconfigured template.
14 . The non-transitory computer readable medium of claim 13 , wherein the identifying of the preconfigured template comprises comparing at least one best match value against a minimum match threshold.
15 . The non-transitory computer readable medium of claim 14 , wherein, when the minimum match threshold is not met, then a generic prompt is used.
16 . The non-transitory computer readable medium of claim 9 , wherein the deriving an answer to the user question is performed without formulating an LLM prompt.
17 . A system for choosing between a large language model agent (LLM agent) or a non-LLM agent, the system comprising:
a storage medium having stored thereon a sequence of instructions; and
one or more processors that execute the sequence of instructions to cause the one or more processors to perform a set of acts, the set of acts comprising,
configuring a content management system (CMS) to expose content objects stored in a shared content repository to a plurality of content management system users;
receiving a user question from a content management system user of the plurality of content management system users;
responsive to receipt of the user question, selecting at least one computing agent from a plurality of candidate computing agents, wherein the plurality of candidate computing agents comprise at least a LLM agent and a non-LLM agent, and the at least one computing agent is selected at least based on results of processing the user question to compare the user question to data stored in a database that represents example questions and is separate from the shared content repository; and
processing the user question by:
when the LLM agent is selected, forming an answer to the user question at least by formulating an LLM prompt using the LLM agent, and sending the LLM prompt to an LLM system, and
when the non-LLM agent is selected, deriving an answer to the user question based on at least characteristics of CMS data associated with at least a subset of the content objects stored in the shared content repository.
18 . The system of claim 17 , further comprising determining a domain-specific query language.
19 . The system of claim 18 , further comprising executing a domain-specific query to produce local query results, the domain-specific query based at least in part on the domain-specific query language.
20 . The system of claim 19 , further comprising reformatting the local query results to render the local query results into natural language.