Crossdomain confidence scoring algorithm using vectors and similarity scores
Aspects of the invention provide a method, computer system, and computer program product for retrieval augmented generation. In one aspect, the method includes receiving a query. The method further includes classifying the query to a first domain within a multitude of domains. The method additionally includes retrieving an index of domain-specific vector embeddings corresponding to the domains. The method further includes prompting a large language model (LLM) with the query and the domain-specific vector embeddings. The method also includes receiving a query response from the LLM as grounded with the most relevant index results. The method further includes forwarding the query response.
1 . A method comprising:
converting each of a plurality of datasets of a plurality of domains into a plurality of domain-specific vector embeddings;
storing the plurality of domain-specific vector embeddings in a vector store of a data repository;
classifying the plurality of datasets into the plurality of domains to obtain a classification;
storing, according to the classification, an index in an index store for each of the plurality of domain-specific vector embeddings;
receiving a query;
classifying, by a recommendation engine of an information retrieval system, the query to a domain within a plurality of domains, wherein classifying the query comprises calculating a weighted confidence score for each domain in the plurality of domains, and wherein the weighted confidence score is calculated as:
weighted
confidence
j
=
∑
i
=
1
k
(
sigmoid
i
j
×
weight
i
j
)
+
η
j
wherein:
j is a domain,
k is a number of top results,
i is a current index,
sigmoid is a sigmoid function,
weight is a confidence score weight, and
ηj is a domain specific biasing constant;
retrieving, by the information retrieval system performing vector search, an index of domain-specific vector embeddings in the plurality of domain-specific vector embeddings of the domain;
transmitting, in an application programming interface (API) call, a prompt to a Large Language Model (LLM), the prompt comprising the query and the domain-specific vector embeddings;
receiving a query response from the LLM as grounded with the most relevant index results; and
forwarding the query response to the user.
2 . The method of claim 1 , further comprising:
evaluating the query response against a criteria to determine that the query response fails the criteria;
transmitting, in a second API call, an adjusted prompt to the LLM, the adjusted prompt comprising the query and adjusted domain-specific vector embeddings of the plurality of domain specific vector embeddings;
receiving the adjusted query response from the LLM, wherein the adjusted query response is forwarded to the user.
3 . The method of claim 1 , wherein classifying the query further comprises:
calculating a global confidence score; and
comparing the plurality of domains and suggesting the domain.
4 . The method of claim 3 , wherein the domain specific biasing constant is calculated as:
η
j
=
ω
c
×
C
j
+
ω
h
×
H
j
+
ω
p
×
P
j
wherein:
ω c is a weight for the current context bias,
C j is a current context relevance to domain j,
ω h is a weight for a conversation history bias,
H j is a conversation history relevance to domain j,
ω p is a weight for a popularity bias, and
P j is a popularity of domain j.
5 . The method of claim 3 , wherein comparing the plurality of domains further comprises:
if the weighted confidence is greater than the global confidence, retrieving results from the domain; and
if the weighted confidence is not greater than the global confidence, retrieving results from other domains in the plurality of domains.
6 . A method comprising:
converting each of a plurality of datasets of a plurality of domains into a plurality of domain-specific vector embeddings;
storing the plurality of domain-specific vector embeddings in a vector store of a data repository;
classifying the plurality of datasets into the plurality of domains to obtain a classification;
storing, according to the classification, an index in an index store for each of the plurality of domain-specific vector embeddings;
receiving a query;
classifying, by a recommendation engine of an information retrieval system, the query to a domain within a plurality of domains,
wherein classifying the query comprises calculating a global confidence for each domain in the plurality of domains, and
wherein the global confidence is calculated as:
global
confidence
=
μ
+
β
+
σ
wherein:
μ is a mean,
β is a tunable hyperparameter, and
σ is a standard deviation;
retrieving, by the information retrieval system performing vector search, an index of domain-specific vector embeddings in the plurality of domain-specific vector embeddings of the domain;
transmitting, in an application programming interface (API) call, a prompt to prompting a Large Language Model (LLM), the prompt comprising the query and the domain-specific vector embeddings;
receiving a query response from the LLM as grounded with the most relevant index results; and
forwarding the query response to the user.
7 . A system comprising:
a computer processor;
memory; and
instructions stored in the memory and executable by the computer processor to cause the computer processor to perform operations, the operations comprising:
converting each of a plurality of datasets of a plurality of domains into a plurality of domain-specific vector embeddings;
storing the plurality of domain-specific vector embeddings in a vector store of a data repository;
classifying the plurality of datasets into the plurality of domains to obtain a classification;
storing, according to the classification, an index in an index store for each of the plurality of domain-specific vector embeddings;
receiving a query;
classifying, by a recommendation engine of an information retrieval system, the query to a domain within a plurality of domains, wherein classifying the query comprises calculating a weighted confidence score for each domain in the plurality of domains, and wherein the weighted confidence score is calculated as:
weighted
confidence
j
=
∑
i
=
1
k
(
sigmoid
i
j
×
weight
i
j
)
+
η
j
wherein:
j is a domain,
k is a number of top results,
i is a current index,
sigmoid is a sigmoid function,
weight is a confidence score weight, and
ηj is a domain specific biasing constant;
retrieving, by the information retrieval system performing vector search, an index of domain-specific vector embeddings in the plurality of domain-specific vector embeddings of the domain;
transmitting, in an application programming interface (API) call, a prompt to a large language model (LLM), the prompt comprising the query and the domain-specific vector embeddings;
receiving a query response from the LLM as grounded with the most relevant index results; and
forwarding the query response.
8 . The system of claim 7 , wherein classifying the query further comprises:
calculating a global confidence score; and
comparing the plurality of domains and suggesting the domain.
9 . The system of claim 8 , wherein the domain specific biasing constant is calculated as:
η
j
=
ω
c
×
C
j
+
ω
h
×
H
j
+
ω
p
×
P
j
wherein:
ω c is a weight for the current context bias,
C j is a current context relevance to domain j,
ω h is a weight for a conversation history bias,
H j is a conversation history relevance to domain j,
ω p is a weight for a popularity bias, and
P j is the popularity of domain j.
10 . The system of claim 8 , wherein the global confidence is calculated as:
global
confidence
=
μ
+
β
+
σ
wherein:
μ is a mean,
β is a tunable hyperparameter, and
σ is a standard deviation.
11 . The system of claim 8 , wherein comparing the plurality of domains further comprises:
if the weighted confidence is greater than the global confidence, retrieving results from the domain; and
if the weighted confidence is not greater than the global confidence, retrieving results from other domains in the plurality of domains.