Enhancing retrieval augmented generation accuracy
Provided is a method including obtaining a prompt, determining a prompt embedding vector representing the prompt in an embedding space, modifying the prompt embedding vector using a trained model configured to adjust prompt embedding vectors to decrease proximity to vectors of blocks in a data set from which data is retrieved to augment generation by the generative AI model, determining that the modified prompt embedding vector is within a threshold distance to vectors in the embedding space corresponding to one or more blocks in the data set, selecting the one or more blocks in the data set, generating a response using the generative AI model based on the selected one or more blocks in the data set, quantifying an amount of influence of the respective block on corresponding text in the generated response, and providing the response and a representation of the quantified amount of influence as an output.
1 . A computer-implemented method, comprising:
obtaining, with a computer system, a prompt that is, or is part of, a request for a generative artificial intelligence (AI) model to generate a response;
determining, with the computer system, a prompt embedding vector representing the prompt in an embedding space, wherein proximity within the embedding space at least partially corresponds to semantic similarity;
modifying, with the computer system, the prompt embedding vector using a trained model configured to adjust prompt embedding vectors to decrease proximity to vectors of blocks of data in a data set from which data is retrieved to augment generation by the generative AI model, wherein the trained model configured to adjust prompt embedding vectors is trained by:
initializing a reinforcement learning environment in which a retrieval system serves as external feedback for evaluating prompt embedding vector transformations;
defining a state space comprising training prompt embedding vectors prior to modification;
defining an action space comprising numerical transformations applied to the training prompt embedding vectors within the embedding space;
defining a reward function that assigns rewards for decreasing proximity between modified training prompt embedding vectors and non-relevant blocks of data while maintaining or increasing proximity to relevant blocks;
iteratively training the model by modifying training prompt embedding vectors and retrieving blocks from a training data set based on similarity to the modified training prompt embedding vectors;
computing a reward signal based on a retrieval quality of selected training blocks relative to the training prompt embedding vectors; and
repeating training steps over multiple prompt-block instances;
determining, with the computer system, that the modified prompt embedding vector is within a threshold distance to one or more vectors in the embedding space corresponding to one or more blocks of data in the data set and, in response, selecting the one or more blocks of data in the data set;
generating, with the computer system, a response using the generative AI model based on the selected one or more blocks of data in the data set;
quantifying, with the computer system, for each of at least one of the selected blocks, an amount of influence of the respective block on corresponding text in the generated response; and
providing, with the computer system, the response and a representation of the quantified amount of influence as an output.
2 . The method of claim 1 , wherein:
the data set comprises a corpus of more than 10,000 natural language text documents;
modifying comprises changing scalars of the prompt embedding vector;
selecting the one or more blocks comprises selecting chunks of the natural language text documents; and
the embedding space has more than 256 dimensions.
3 . The computer-implemented method of claim 1 , wherein the generated response is also based on the prompt.
4 . The computer-implemented method of claim 1 , wherein the generated response is also based on the modified prompt embedding vector.
5 . The computer-implemented method of claim 1 , wherein quantifying the amount of influence of the respective block comprises calculation of at least one of the following: a model loss associated with the respective block to be removed from the selected blocks of data, an attention weight from the respective data block, a log-odds score, or a likelihood ratio.
6 . The computer-implemented method of claim 1 , further comprising:
creating, with the computer system, a representation of modifications applied to the prompt embedding vector;
inputting, with the computer system, the representation of the modifications to the generative AI model; and
generating, with the computer system, a natural language explanation describing the modifications made to the prompt embedding vector using the generative AI model.
7 . The computer-implemented method of claim 1 , further comprising regenerating the modified prompt embedding vector in response to determining that the response fails to meet a fidelity threshold.
8 . The computer-implemented method of claim 1 , wherein quantifying the influence of each selected block on the generated response comprises:
determining, with the computer system, a numerical representation of each selected block by mapping the text of the block to a multi-dimensional vector space;
determining, with the computer system, a numerical representation of the generated response by mapping the text of the response to the multi-dimensional vector space;
associating, with the computer system, individual portions of the generated response with corresponding portions of the selected blocks based on the proximity of their numerical representations within the vector space;
computing, with the computer system, a measure of contribution for each selected block by aggregating proximity values between its numerical representation and the numerical representations of the associated portions of the generated response; and
normalizing, with the computer system, the measure of contribution across all selected blocks.
9 . The computer-implemented method of claim 1 , wherein the provided representation of the quantified amount of influence comprises:
associations between selected blocks and corresponding portions of the generated response; and
a value corresponding to each of the selected blocks, wherein each value corresponding to selected blocks represents a measure of contribution of the respective selected block to the corresponding portion of the response.
10 . The computer-implemented method of claim 1 , wherein modifying the prompt embedding vector comprises providing a reformulated query using a natural language processing model trained based on retrieval accuracy.
11 . The computer-implemented method of claim 1 , wherein the data set is indexed by computing embedding vectors in the embedding space of text blocks using a hierarchical navigable small world graph.
12 . The computer-implemented method of claim 1 , wherein providing the representation of the quantified influence comprises providing data or instructions used to form a graphical user interface mapping response text segments to corresponding retrieved sources using a provenance-tracking visualization.
13 . The computer-implemented method of claim 1 , wherein modifying the prompt embedding vector comprises adjusting the vector representation based on stored session history data.
14 . The computer-implemented method of claim 1 , wherein selecting blocks comprises federated retrieval across corpora.
15 . The computer-implemented method of claim 1 , further comprising reranking the selected blocks based on proximity to the prompt embedding vector in the embedding space before generating the response.
16 . The computer-implemented method of claim 1 , wherein the method is implemented in a healthcare AI system, wherein retrieved blocks comprise medical records, clinical guidelines, research articles, or diagnostic data, and wherein the generated response includes at least one of: patient-specific recommendations based on retrieved clinical data, summaries of medical literature with ranked provenance scores, diagnostic support by correlating patient symptoms with retrieved cases, or drug interaction analysis based on pharmaceutical and clinical trial data.
17 . The computer-implemented method of claim 1 , wherein the quantifying for each of at least one of the selected blocks comprises steps for quantifying the blocks.
18 . The computer-implemented method of claim 1 , wherein the modifying the prompt embedding vector comprises steps for adjusting prompt embedding vectors to decrease proximity to vectors of blocks in a data set.
19 . A computer-implemented method, comprising:
obtaining, with a computer system, a prompt that is, or is part of, a request for a generative artificial intelligence (AI) model to generate a response;
determining, with the computer system, a prompt embedding vector representing the prompt in an embedding space, wherein proximity within the embedding space at least partially corresponds to semantic similarity;
modifying, with the computer system, the prompt embedding vector using a trained model configured to adjust prompt embedding vectors to decrease proximity to vectors of blocks of data in a data set from which data is retrieved to augment generation by the generative AI model;
determining, with the computer system, that the modified prompt embedding vector is within a threshold distance to one or more vectors in the embedding space corresponding to one or more blocks of data in the data set and, in response, selecting the one or more blocks of data in the data set;
generating, with the computer system, a response using the generative AI model based on the selected one or more blocks of data in the data set;
quantifying, with the computer system, for each of at least one of the selected blocks, an amount of influence of the respective block on corresponding text in the generated response;
providing, with the computer system, the response and a representation of the quantified amount of influence as an output;
creating, with the computer system, a representation of modifications applied to the prompt embedding vector;
inputting, with the computer system, the representation of the modifications to the generative AI model; and
generating, with the computer system, a natural language explanation describing the modifications made to the prompt embedding vector using the generative AI model.
20 . The computer-implemented method of claim 19 , wherein the generated response is also based on the prompt.
21 . The computer-implemented method of claim 19 , wherein the generated response is also based on the modified prompt embedding vector.
22 . The computer-implemented method of claim 19 , wherein quantifying the amount of influence of the respective block comprises calculation of at least one of the following: a model loss associated with the respective block to be removed from the selected data blocks, an attention weight from the respective block of data, a log-odds score, or a likelihood ratio.
23 . The computer-implemented method of claim 19 , further comprising regenerating the modified prompt embedding vector in response to determining that the response fails to meet a fidelity threshold.
24 . The computer-implemented method of claim 19 , wherein providing the representation of the quantified influence comprises providing data or instructions used to form a graphical user interface mapping response text segments to corresponding retrieved sources using a provenance-tracking visualization.
25 . The computer-implemented method of claim 19 , wherein the method is implemented in a healthcare AI system, wherein retrieved blocks comprise medical records, clinical guidelines, research articles, or diagnostic data, and wherein the generated response includes at least one of: patient-specific recommendations based on retrieved clinical data, summaries of medical literature with ranked provenance scores, diagnostic support by correlating patient symptoms with retrieved cases, or drug interaction analysis based on pharmaceutical and clinical trial data.
26 . A computer-implemented method, comprising:
obtaining, with a computer system, a prompt that is, or is part of, a request for a generative artificial intelligence (AI) model to generate a response;
determining, with the computer system, a prompt embedding vector representing the prompt in an embedding space, wherein proximity within the embedding space at least partially corresponds to semantic similarity;
modifying, with the computer system, the prompt embedding vector using a trained model configured to adjust prompt embedding vectors to decrease proximity to vectors of blocks of data in a data set from which data is retrieved to augment generation by the generative AI model;
determining, with the computer system, that the modified prompt embedding vector is within a threshold distance to vectors in the embedding space corresponding to one or more blocks of data in the data set and, in response, selecting the one or more blocks of data in the data set;
generating, with the computer system, a response using the generative AI model based on the selected one or more blocks of data in the data set;
quantifying, with the computer system, for each of at least one of the selected blocks, an amount of influence of the respective block on corresponding text in the generated response, the quantifying comprising:
determining, with the computer system, a numerical representation of each selected block by mapping the text of the block to a multi-dimensional vector space;
determining, with the computer system, a numerical representation of the generated response by mapping the text of the response to the multi-dimensional vector space;
associating, with the computer system, individual portions of the generated response with corresponding portions of the selected blocks based on the proximity of their numerical representations within the vector space;
computing, with the computer system, a measure of contribution for each selected block by aggregating proximity values between its numerical representation and the numerical representations of the associated portions of the generated response; and
normalizing, with the computer system, the measure of contribution across all selected blocks; and
providing, with the computer system, the response and a representation of the quantified amount of influence as an output.
27 . The computer-implemented method of claim 26 , wherein the provided representation of the quantified amount of influence comprises:
associations between selected blocks and corresponding portions of the generated response; and
a value corresponding to each of the selected blocks, wherein each value corresponding to selected blocks represents a measure of contribution of the respective selected block to the corresponding portion of the response; and
wherein the multi-dimensional vector space is the embedding space.
28 . The method of claim 26 , wherein:
the data set comprises a corpus of more than 10,000 natural language text documents;
modifying comprises changing scalars of the prompt embedding vector;
selecting the one or more blocks comprises selecting chunks of the natural language text documents; and
the embedding space has more than 256 dimensions.
29 . The computer-implemented method of claim 26 , wherein quantifying the amount of influence of the respective block comprises calculation of at least one of the following: a model loss associated with the respective block to be removed from the selected data blocks, an attention weight from the respective block of data, a log-odds score, or a likelihood ratio.
30 . The computer-implemented method of claim 26 , further comprising reranking the selected blocks based on proximity to the prompt embedding vector in the embedding space before generating the response.
31 . The computer-implemented method of claim 26 , wherein the quantifying for each of at least one of the selected blocks comprises steps for quantifying the blocks.
32 . The computer-implemented method of claim 26 , wherein providing the representation of the quantified influence comprises providing data or instructions used to form a graphical user interface mapping response text segments to corresponding retrieved sources using a provenance-tracking visualization.
33 . A computer-implemented method, comprising:
obtaining, with a computer system, a prompt that is, or is part of, a request for a generative artificial intelligence (AI) model to generate a response;
determining, with the computer system, a prompt embedding vector representing the prompt in an embedding space, wherein proximity within the embedding space at least partially corresponds to semantic similarity;
modifying, with the computer system, the prompt embedding vector using a trained model configured to adjust prompt embedding vectors to decrease proximity to vectors of blocks of data in a data set from which data is retrieved to augment generation by the generative AI model;
determining, with the computer system, that the modified prompt embedding vector is within a threshold distance to vectors in the embedding space corresponding to one or more blocks of data in the data set and, in response, selecting the one or more blocks of data in the data set;
generating, with the computer system, a response using the generative AI model based on the selected one or more blocks of data in the data set;
quantifying, with the computer system, for each of at least one of the selected blocks, an amount of influence of the respective block on corresponding text in the generated response; and
providing, with the computer system, the response and a representation of the quantified amount of influence as an output, wherein the provided representation of the quantified amount of influence comprises:
associations between selected blocks and corresponding portions of the generated response; and
a value corresponding to each of the selected blocks, wherein each value corresponding to selected blocks represents a measure of contribution of the respective selected block to the corresponding portion of the response.
34 . The computer-implemented method of claim 33 , wherein the generated response is also based on the prompt.
35 . The computer-implemented method of claim 33 , wherein the generated response is also based on the modified prompt embedding vector.
36 . The computer-implemented method of claim 33 , wherein quantifying the amount of influence of the respective block comprises calculation of at least one of the following: a model loss associated with the respective block to be removed from the selected blocks of data, an attention weight from the respective data block, a log-odds score, or a likelihood ratio.
37 . The computer-implemented method of claim 33 , further comprising regenerating the modified prompt embedding vector in response to determining that the response fails to meet a fidelity threshold.
38 . The computer-implemented method of claim 33 , wherein providing the representation of the quantified influence comprises providing data or instructions used to form a graphical user interface mapping response text segments to corresponding retrieved sources using a provenance-tracking visualization.
39 . The computer-implemented method of claim 33 , wherein the method is implemented in a healthcare AI system, wherein retrieved blocks comprise medical records, clinical guidelines, research articles, or diagnostic data, and wherein the generated response includes at least one of: patient-specific recommendations based on retrieved clinical data, summaries of medical literature with ranked provenance scores, diagnostic support by correlating patient symptoms with retrieved cases, or drug interaction analysis based on pharmaceutical and clinical trial data.
40 . The computer-implemented method of claim 33 , wherein:
the data set comprises a corpus of more than 10,000 natural language text documents;
modifying comprises changing scalars of the prompt embedding vector;
selecting the one or more blocks comprises selecting chunks of the natural language text documents; and
the embedding space has more than 256 dimensions.