Content retrieval based on a generative AI response
Systems and methods are described for performing retrieval of information based on a generative AI prompt and response. A system can receive a prompt from a user, then generate a response to the prompt by using a generative AI model. The system may then determine a span of text within the response, which may be a portion of text from the response to be used as the basis for a retrieval or search with respect to one or more data repositories. The span of text, response, and prompt can be used to perform a search to retrieve results, where the span of text may be used as a search term in the search and the prompt and response may be used as context for ranking during the search. The results can be presented to the user to be compared against the prompt and response.
1 . A system comprising:
memory; and
at least one computing device configured with computer-executable instructions that, when executed, cause the at least one computing device to:
receive, from a user computing device via user interaction with a user interface (UI), a prompt as natural language text input;
generate, via a large language model (LLM), a response based on the prompt, wherein the response comprises text generated by the LLM;
send, to the user computing device, the response to be presented within the UI;
subsequent to presentation by the user computing device of the response within the UI, identify, based on the prompt and the response, a span of text within the response to perform a search associated with the span of text, wherein the span of text is determined by,
subsequent to presentation of the response within the UI, a user of the user computing device selecting the span of text from the text of the response;
retrieve, from one or more data repositories, retrieval results using the span of text as a retrieval term, wherein the retrieval results each comprise a portion of text from information associated with the one or more data repositories that includes or relates to the span of text;
rank the retrieval results based at least in part on context determined from the prompt and the response generated by the LLM;
select one or more highest ranked retrieval results for presentation within the UI; and
send, to the user computing device, the one or more highest ranked retrieval results to be displayed.
2 . The system of claim 1 , wherein the results are displayed for the user within the UI along with the prompt and the response.
3 . The system of claim 1 , wherein the UI is a first UI and the results are displayed for the user within a second UI separate from the first UI.
4 . The system of claim 1 , wherein the prompt is a question related to a product, the span relates to a specific feature of the product, and the one or more highest ranked retrieval results comprise portions of one or more of (a) an item listing from an electronic catalog, (b) a review of the product, (c) a question and answer page regarding the product, or seller information regarding a seller of the product.
5 . A computer-implemented method comprising:
receiving, from a computing device, a prompt as input to a generative artificial intelligence (AI) model;
receiving, from the generative AI model, a response based on the prompt;
sending the response to the computing device;
subsequent to presentation by the computing device of the response within a user interface (UI), identifying, based on the prompt and the response, a span of text within the response to perform a search associated with the span of text, wherein the span of text is determined by,
subsequent to presentation of the response within the UI, a user of the computing device selecting the span of text from the response;
performing a retrieval of one or more data repositories to identify content related to the span of text, wherein retrieval results are selected based at least in part on context determined from text of the prompt and text of the response; and
sending, to the computing device, the retrieval results to be displayed.
6 . The computer-implemented method of claim 5 , wherein the span of text is automatically identified based in part by performing Term Frequency Inverse Document Frequency (TF-IDF) analysis using frequency of words from the response relative to word frequency in a large set of responses previously generated by the generative AI model.
7 . The computer-implemented method of claim 5 , wherein the span of text is automatically identified by determining that the span of text was below a confidence score threshold, wherein a confidence score associated with the span of text was generated by the generative AI model.
8 . The computer-implemented method of claim 5 , wherein the span of text is automatically identified by determining that the generative AI model flagged the span of text for verification.
9 . The computer-implemented method of claim 5 , wherein the span of text is automatically identified by identifying patterns of prior prompts, responses, and manually selected spans of text using a training dataset.
10 . The computer-implemented method of claim 5 , wherein the context is determined by performing Term Frequency Inverse Document Frequency (TF-IDF) analysis to identify relevant contextual text from within the prompt and response.
11 . The computer-implemented method of claim 5 , further comprising:
automatically identifying the span of text and prior to retrieving the results;
receiving, from the computing device, a modified version of the span of text manually modified by a user of the computing device after automatically identifying the span of text; and
updating the span of text to the modified span of text.
12 . The computer-implemented method of claim 5 , wherein the retrieval is performed by:
assigning embeddings to the span of text and context with weights, wherein the span of text is weighted more than the context; and
using the weighted embedding to perform a search.
13 . The computer-implemented method of claim 5 , wherein the results are displayed for the user within a first UI along with the prompt and the response.
14 . The computer-implemented method of claim 13 , wherein the results are displayed for the user within a second UI separate from the first UI.
15 . One or more non-transitory computer-readable media comprising computer-executed instructions that, when executed by a computing system, cause the computing system to:
receive, from a computing device, a prompt as input to a generative artificial intelligence (AI) model;
receive, from the generative AI model, a response based on the prompt;
send the response to the computing device;
subsequent to presentation by the computing device of the response within a user interface (UI), identify, based on the prompt and the response, a span of text within the response to perform a search associated with the span of text, wherein the span of text is determined by, subsequent to presentation of the response within the UI, a user of the computing device selecting the span of text from the text of the response;
perform a retrieval of one or more data repositories to identify content related to the span of text, wherein retrieval results are selected based at least in part on context determined from text of the prompt and text of the response; and
send, to the computing device, the retrieval results to be displayed.
16 . The one or more non-transitory computer-readable media of claim 15 , wherein the context is determined by performing Term Frequency Inverse Document Frequency (TF-IDF) analysis to identify relevant contextual text from within the prompt and response.
17 . The one or more non-transitory computer-readable media of claim 15 , wherein the retrieval is performed at least in part by:
generating one or more embeddings representing (a) the span of text and (b) other text from the response and prompt, wherein the span of text is weighted more than the other text; and
using the one or more embeddings as input to perform the retrieval.
18 . The one or more non-transitory computer-readable media of claim 15 , wherein the span of text is automatically identified based in part by performing Term Frequency Inverse Document Frequency (TF-IDF) analysis using frequency of words from the response relative to word frequency in a large set of responses previously generated by the generative AI model.
19 . The one or more non-transitory computer-readable media of claim 15 , wherein the span of text is automatically identified by determining that the span of text was below a confidence score threshold, wherein a confidence score associated with the span of text was generated by the generative AI model.
20 . The one or more non-transitory computer-readable media of claim 15 , wherein the results are displayed for a user of the computing device within a single user interface (UI) along with the prompt and the response.