Responding to query using expert augmentation of retrieval-augmented generation
A system includes a processor that executes computer executable components stored in a memory. The computer executable components can include a retrieval-augmented generation component linked to multiple knowledge base indexes and a reception component that receives a query and instructs the retrieval-augmented generation component to search a first knowledge base index of relevant documents. The computer executable components can include a question component that instructs the retrieval-augmented generation component to search a second knowledge base index of relevant expert question-answer pairs, and an expert identification component that instructs the retrieval-augmented generation component to search a third knowledge base index of expert profiles, identifies an at least one knowledge domain expert, and gathers input from the at least one expert. The computer executable components can include a response component that, upon a determination that there is sufficient information for a satisfactory response to the query, generates a response to the query.
1 . A system, comprising:
a memory configured to store computer executable components; and
a processor that executes at least one of the computer executable components that:
accesses multiple knowledge base indexes comprising a document vector store that stores embeddings for documents of a knowledge base, an expert question-answer vector store that stores embeddings for expert question-answer pairs that could not be answered based on the knowledge base, and an expert profiles vector store that stores embeddings for expert profiles;
receives a query; and
in response to receiving the query:
performs a first semantic search over the document vector store using a first knowledge base index of relevant documents,
determines whether there is sufficient information resulting from the first semantic search to satisfactorily respond to the query according to a defined criterion,
based on determining that the first semantic search did not result in sufficient information to satisfactorily respond to the query according to the defined criterion, performs a second semantic search over the expert question-answer vector store using a second knowledge base index of relevant expert question-answer pairs,
determines whether there is sufficient information resulting from the second semantic search to satisfactorily respond to the query according to the defined criterion,
based on determining that the second semantic search did not result in sufficient information to satisfactorily respond to the query according to the defined criterion, performs a third semantic search over the expert profiles vector store using a third knowledge base index of expert profiles, wherein performing the third semantic search comprises:
identifying at least one knowledge domain expert,
gathering input from the at least one knowledge domain expert, wherein the input comprises one or more expert-recommended documents not yet present in the document vector store,
storing the one or more expert-recommended documents in a staging document vector store that is separate from the document vector store, and
performing a staged retrieval-augmented generation using only the staging document vector store to attempt to generate sufficient information to satisfactorily respond to the query according to the defined criterion,
determining whether there is sufficient information resulting from the third semantic search to satisfactorily respond to the query according to the defined criterion,
upon a determination that any results of the first semantic search, the second semantic search, and the third semantic search resulted in sufficient information to satisfactorily respond to the query according to the defined criterion, generates a response to the query, and
based on a determination that the staged retrieval-augmented generation resulted in sufficient information to satisfactorily respond to the query according to the defined criterion, promotes the one or more expert-recommended documents from the staging document vector store into the document vector store for use in responding to subsequent queries.
2 . The system of claim 1 , wherein the at least one of the computer executable components further:
receives feedback from a user regarding sufficiency of the response to the query.
3 . The system of claim 2 , wherein the at least one of the computer executable components further:
updates at least one of the document vector store, the expert question-answer vector store, or the expert profiles vector store based on the received feedback.
4 . The system of claim 1 , wherein the at least one of the computer executable components further:
gathers additional input from guardrails models or knowledge content owners to evaluate sufficiency of the first knowledge base index.
5 . The system of claim 4 , wherein the additional input pertains to at least one of adherence to organizational policies, ethical standards, or regulatory requirements.
6 . The system of claim 4 , wherein the at least one of the computer executable components further:
identifies inconsistencies, redundancies, or inaccuracies in at least one of the document vector store, the expert question-answer vector store, or the expert profiles vector store and updates the at least one of the document vector store, the expert question-answer vector store, or the expert profiles vector store.
7 . The system of claim 1 , wherein the at least one of the computer executable components further:
updates at least one of the document vector store, the expert question-answer vector store, or the expert profiles vector store based on additional input from a respective knowledge owner.
8 . The system of claim 1 , wherein the at least one of the computer executable components further:
identifies the at least one knowledge domain expert based on qualifications or contextual information stored in the expert profiles vector store.
9 . The system of claim 8 , wherein the qualifications further comprise a degree to which a knowledge domain expert's associated profile aligns with a subject matter or content of the query.
10 . The system of claim 9 , wherein the degree to which the knowledge domain expert's associated profile aligns with the subject matter or content of the query is determined by at least one of: the knowledge domain expert's expertise area, experience, or relevant certifications.
11 . The system of claim 8 , wherein the contextual information further comprises at least one of: historical responsiveness of the expert to prior queries, availability of the expert, or urgency of the query.
12 . A computer-implemented method, comprising:
accessing, by a system comprising a processor, multiple knowledge base indexes comprising a document vector store that stores embeddings for documents of a knowledge base, an expert question-answer vector store that stores embeddings for expert question-answer pairs that could not be answered based on the knowledge base, and an expert profiles vector store that stores embeddings for expert profiles,
receiving, by the system, a query; and
in response to receiving the query:
performing, by the system, a first semantic search over the document vector store using a first knowledge base index of relevant documents,
determining, by the system, whether there is sufficient information resulting from the first semantic search to satisfactorily respond to the query according to a defined criterion,
based on determining that the first semantic search did not result in sufficient information to satisfactorily respond to the query according to the defined criterion, performing, by the system, a second semantic search over the expert question-answer vector store using a second knowledge base index of relevant expert question-answer pairs;
determining, by the system, whether there is sufficient information resulting from the second semantic search to satisfactorily respond to the query according to the defined criterion,
based on determining that the second semantic search did not result in sufficient information to satisfactorily respond to the query according to the defined criterion, performing, by the system, a third semantic search over the expert profiles vector store using a third knowledge base index of expert profiles, wherein performing the third semantic search comprises:
identifying at least one knowledge domain expert,
gathering input from the at least one knowledge domain expert wherein the input comprises one or more expert-recommended documents not yet present in the document vector store,
storing the one or more expert-recommended documents in a staging document vector store that is separate from the document vector store, and
performing a staged retrieval-augmented generation using only the staging document vector store to attempt to generate sufficient information to satisfactorily respond to the query according to the defined criterion,
determining, by the system, whether there is sufficient information resulting from the third semantic search to satisfactorily respond to the query according to the defined criterion,
upon a determination that any results of the first semantic search, the second semantic search, and the third semantic search resulted in sufficient information satisfactorily respond to the query according to the defined criterion, generating, by the system, a response to the query, and
based on a determination that the staged retrieval-augmented generation resulted in sufficient information to satisfactorily respond to the query according to the defined criterion, promoting, by the system, the one or more expert-recommended documents from the staging document vector store into the document vector store for use in responding to subsequent queries.
13 . The method of claim 12 , further comprising receiving, by the system, feedback from a user regarding sufficiency of the response to the query.
14 . The computer-implemented method of claim 13 , further comprising updating, by the system, at least one of the document vector store, the expert question-answer vector store, or the expert profiles vector store based on the received feedback.
15 . The computer-implemented method of claim 12 , further comprising gathering, by the system, additional input from guardrails models or knowledge content owners to evaluate sufficiency of the first knowledge base index, wherein the additional input pertains to at least one of adherence to organizational policies, ethical standards, or regulatory requirements.
16 . A computer program product comprising a non-transitory computer readable storage having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:
access multiple knowledge base indexes comprising a document vector store that stores embeddings for documents of a knowledge base, an expert question-answer vector store that stores embeddings for expert question-answer pairs that could not be answered based on the knowledge base, and an expert profiles vector store that stores embeddings for expert profiles;
receive a query; and
in response to receiving the query:
perform a first semantic search over the document vector store using a first knowledge base index of relevant documents,
determine whether there is sufficient information resulting from the first semantic search to satisfactorily respond to the query according to a defined criterion,
based on determining that the first semantic search did not result in sufficient information to satisfactorily respond to the query according to the defined criterion, perform a second semantic search over the expert question-answer vector store using a second knowledge base index of relevant expert question-answer pairs;
determine whether there is sufficient information resulting from the second semantic search to satisfactorily respond to the query according to the defined criterion,
based on determining that the second semantic search did not result in sufficient information to satisfactorily respond to the query according to the defined criterion, perform a third semantic search over the expert profiles vector store using a third knowledge base index of expert profiles, wherein performing the third semantic search comprises:
identifying at least one knowledge domain expert,
gathering input from the at least one knowledge domain expert wherein the input comprises one or more expert-recommended documents not yet present in the document vector store,
storing the one or more expert-recommended documents in a staging document vector store that is separate from the document vector store, and
performing a staged retrieval-augmented generation using only the staging document vector store to attempt to generate sufficient information to satisfactorily respond to the query according to the defined criterion,
determine whether there is sufficient information resulting from the third semantic search to satisfactorily respond to the query according to the defined criterion,
upon a determination that any results of the first semantic search, the second semantic search, and the third semantic search resulted in sufficient information satisfactorily respond to the query according to the defined criterion, generate a response to the query, and
based on a determination that the staged retrieval-augmented generation resulted in sufficient information to satisfactorily respond to the query according to the defined criterion, promote the one or more expert-recommended documents from the staging document vector store into the document vector store for use in responding to subsequent queries.
17 . The computer program product of claim 16 , wherein the program instructions are executable by the processor to cause the processor to receive feedback from a user regarding sufficiency of the response to the query.
18 . The computer program product of claim 17 , wherein the program instructions are executable by the processor to cause the processor to update at least one of the document vector store, the expert question-answer vector store, or the expert profiles vector store based on the received feedback.
19 . The computer program product of claim 16 , wherein the program instructions are executable by the processor to cause the processor to gather additional input from guardrails models or knowledge content owners to evaluate sufficiency of the first knowledge base index.
20 . The computer program product of claim 19 , wherein the additional input pertains to at least one of adherence to organizational policies, ethical standards, or regulatory requirements.