Granted Patent B1

US 12,579,174 · App. 18/621,221 · Granted Mar 17, 2026

Inventors: Nicolaas Jedema (Santa Barbara, CA); Leonardo Filipe Rodrigues Ribeiro (Kirkland, WA); Alessandro Moschitti (Playa de Rey, CA); Matteo Gabburo (Bassano del Grappa, CA); Siddhant Garg (San Jose, CA)

Assignee: Amazon Technologies, Inc.

G06F16/3329G06F16/338G06F40/40

Abstract

Claims (115)

1 . A computer-implemented method comprising:

receiving first data representing a first question;

receiving second data representing a reference answer to the first question;

sending a first request including the first question to a first component configured to provide data usable by a language model to generate a response;

receiving, in response to the first request and from the first component, a first plurality of document results;

processing the first data, the second data, and the first plurality of document results using a first trained model to determine a first value representing whether at least one document result of the first plurality of document results can be used to determine an answer that corresponds to the reference answer;

determining that the first value fails to satisfy a condition;

receiving third data representing a user query;

determining, using the third data, that the user query corresponds to the first question;

based on the first value failing to satisfy the condition, sending, to a second component, a second request including the first question, the second component configured to provide data usable by the language model to generate a response;

receiving, from the second component, a second plurality of document results;

processing, using the language model, the third data and the second plurality of document results to generate a response to the user query; and

causing presentation of the response.

2 . The computer-implemented method of claim 1 , wherein the second data includes a first set of sentences and the method further comprises:

processing, using the first trained model, a first document result of the first plurality of document results and the second data to determine a second value representing that the first document result can be used to determine the reference answer, the second value being based on a first portion of the first set of sentences being semantically similar to a second set of sentences in the first document result;

processing, using the first trained model, a second document result of the first plurality of document results and the reference answer to determine a third value representing that the second document result can be used to determine the reference answer, the third value being based on a second portion of the first set of sentences being semantically similar to a third set of sentences in the second document result; and

determining the first value based on the second value and the third value.

3 . The computer-implemented method of claim 1 , further comprising:

processing, using the first trained model, the first plurality of document results and the reference answer to determine:

a second value representing that a first document result of the first plurality of document results includes a first set of tokens semantically similar to a first portion of the reference answer, and

a third value representing that a second document result of the first plurality of document results includes a second set of tokens semantically similar to a second portion of the reference answer; and

determining the first value based on the second value and third value.

4 . The computer-implemented method of claim 1 , further comprising:

receiving fourth data including a second question and a corresponding second answer;

receiving, from the first component, a third plurality of document results based on a third request including the second question;

processing, using the first trained model, the fourth data and the third plurality of document results to determine a second value representing the third plurality of document results can be used to determine the second answer;

determining that the second value satisfies the condition;

based on the second value satisfying the condition, storing, in a data storage, fifth data including the third plurality of document results, the fifth data associated with the second question;

receiving a second user query corresponding to the second question;

determining, from the data storage, the fifth data;

determining a prompt including the second user query and the fifth data;

processing, using the language model, the prompt to generate a second response to the second user query; and

causing presentation of the second response.

5 . A computer-implemented method comprising:

receiving first question data representing a first question;

receiving first answer data representing a first answer corresponding to the first question;

receiving a first plurality of document results based on a first request to a first component, the first request including the first question data;

processing, using a trained model, the first plurality of document results, the first question data and the first answer data to determine a first value representing whether at least one of the first plurality of document results can be used to determine a first response that corresponds to the first answer;

determining that the first value fails to satisfy a first condition; and

based on the first value failing to satisfy the first condition, causing a language model to use a second plurality of document results to generate the first response, wherein the second plurality of document results are determined using a second component.

6 . The computer-implemented method of claim 5 , further comprising:

determining, using the trained model, a second value representing that a first document result of the first plurality of document results corresponds to the first answer;

and

determining the first value based on the second value.

7 . The computer-implemented method of claim 5 , further comprising:

determining, using the trained model, a second value representing that a first document result of the first plurality of document results corresponds to a first portion of the first answer;

determining, using the trained model, a third value representing that a second document result of the first plurality of document results corresponds to a second portion of the first answer, the second portion being different than the first portion; and

determining the first value based on the second value and the third value.

8 . The computer-implemented method of claim 5 , further comprising:

processing, using the trained model, the second plurality of document results, the first question data and the first answer data to determine a second value representing that at least one of the second plurality of document results can be used to determine the first response corresponding to the first answer;

after determining that the second value satisfies the first condition, storing data representing an association between the second plurality of document results and the first question data;

receiving a user query corresponding to the first question;

based on the user query corresponding to the first question, retrieving the second plurality of document results;

determining a prompt based on the user query and including the second plurality of document results; and

processing, using the language model, the prompt to determine the first response to the user query.

9 . The computer-implemented method of claim 5 , further comprising:

receiving second question data representing a second question;

receiving second answer data representing a second answer corresponding to the second question;

receiving a third plurality of document results based on a second request to the first component, the second request including the second question data;

processing, using the trained model, the third plurality of document results, the second question data and the second answer data to determine a second value representing whether at least one of the second plurality of document results can be used to determine a second response that corresponds to the second answer;

determining that the second value fails to satisfy the first condition; and

based on the second value failing to satisfy the first condition, causing the language model to generate a third response to an input corresponding to the second question, the third response indicating that the language model is unable to respond to the input.

10 . The computer-implemented method of claim 5 , further comprising:

receiving second answer data representing a second answer generated by the language model to the first question;

processing, using the trained model, the first plurality of document results, the first question data and the second answer data to determine a second value representing that at least one of the first plurality of document results can be used to determine the second answer;

determining the second value fails to satisfy the first condition; and

based on the second value failing to satisfy the first condition, determining that the language model is to be further trained with respect to the first question.

11 . The computer-implemented method of claim 5 , further comprising:

storing data representing an association between the first question data and the second component, the association indicating that the second component to be used to retrieve content for an input similar to the first question data;

receiving user input data that is semantically similar to the first question data;

determining, based on the stored data, that the second component is to be used;

determining a prompt based on the user input data and including an identifier for the second component; and

processing, using the language model, the prompt to generate a request to the second component.

12 . A system comprising:

at least one processor; and

at least one memory including instructions that, when executed by the at least one processor, cause the system to:

receive first question data representing a first question;

receive first answer data representing a first answer corresponding to the first question;

receive a first plurality of document results based on a first request to a first component, the first request including the first question data;

process, using a trained model, the first plurality of document results, the first question data and the first answer data to determine a first value representing that at least one of the first plurality of document results can be used to determine a first response that corresponds to the first answer;

determine that the first value fails to satisfy a first condition; and

based on the first value failing to satisfy the first condition, cause a language model to use a second plurality of document results to generate the first response, wherein the second plurality of document results are determined using a second component.

13 . The system of claim 12 , wherein the at least one memory includes further instructions that, when executed by the at least one processor, further cause the system to:

determine, using the trained model, a second value representing that a first document result of the first plurality of document results corresponds to the first answer;

and

determine the first value based on the second value.

14 . The system of claim 12 , wherein the at least one memory includes further instructions that, when executed by the at least one processor, further cause the system to:

determine, using the trained model, a second value representing that a first document result of the first plurality of document results corresponds to a first portion of the first answer;

determine, using the trained model, a third value representing that a second document result of the first plurality of document results corresponds to a second portion of the first answer, the second portion being different than the first portion; and

determine the first value based on the second value and the third value.

15 . The system of claim 12 , wherein the at least one memory includes further instructions that, when executed by the at least one processor, further cause the system to:

process, using the trained model, the second plurality of document results, the first question data and the first answer data to determine a second value representing that at least on of the second plurality of document results can be used to determine the first response corresponding to the first answer;

after determining that the second value satisfies the first condition, store data representing an association between the second plurality of document results and the first question data;

receive a user query corresponding to the first question;

based on the user query corresponding to the first question, retrieve the second plurality of document results;

determine a prompt based on the user query and including the second plurality of document results; and

process, using the language model, the prompt to determine the first response to the user query.

16 . The system of claim 12 , wherein the at least one memory includes further instructions that, when executed by the at least one processor, further cause the system to:

receive second question data representing a second question;

receive second answer data representing a second answer corresponding to the second question;

receive a third plurality of document results based on a second request to the first component, the second request including the second question data;

process, using the trained model, the third plurality of document results, the second question data and the second answer data to determine a second value representing whether at least one of the second plurality of document results can be used to determine a second response that corresponds to the second answer;

determine that the second value fails to satisfy the first condition; and

based on the second value failing to satisfy the first condition, cause the language model to generate a third response to an input corresponding to the second question, the third response indicating that the language model is unable to respond to the input.

17 . The system of claim 12 , wherein the at least one memory includes further instructions that, when executed by the at least one processor, further cause the system to:

receive second answer data representing a second answer generated by the language model to the first question;

process, using the trained model, the first plurality of document results, the first question data and the second answer data to determine a second value representing that at least one of the first plurality of document results can be used to determine the second answer;

determine the second value fails to satisfy the first condition; and

based on the second value failing to satisfy the first condition, determine that the language model is to be further trained with respect to the first question.

18 . The system of claim 12 , wherein the at least one memory includes further instructions that, when executed by the at least one processor, further cause the system to:

store data representing an association between the first question data and the second component, the association indicating that the second component to be used to retrieve content for an input similar to the first question data;

receive user input data that is semantically similar to the first question data;

determine, based on the stored data, that the first component is to be used;

determine a prompt based on the user input data and including an identifier for the second component; and

process, using the language model, the prompt to generate a request to the second component.

Assignments (1)

References Cited (96)