Granted Patent B1

US 12,505,288 · App. 18/536,917 · Granted Dec 23, 2025

Inventors: Haibo Ding (Fremont, CA); Panpan Xu (Santa Clara, CA); Huan Song (San Jose, CA); James Robert Golden (Oakland, CA); Yawei Wang (Santa Clara, CA); Lin Lee Cheong (Redwood City, CA)

Assignee: Amazon Technologies, Inc.

G06F40/20

Abstract

Claims (85)

1 . A system comprising:

one or more storage media storing instructions; and

one or more processors configured to execute the instructions to cause the system to: receive a first prompt from a user device to be input to a large language model (LLM);

generate, by at least using a first machine learning (ML) model, a classification of the first prompt;

select, based at least in part on the classification, a second ML model from a plurality of ML models, the second ML model configured to output first LLM prompts based at least in part on a set of templates, the plurality of ML models comprising a third ML model associated with a different classification and configured to output second LLM prompts independently of any template in the set of templates;

provide a first input to the second ML model based at least in part on the first prompt and the classification;

determine a second prompt that is output by the second ML model based at least in part on the first input and a template associated with the classification;

provide the second prompt to the LLM;

receive a first response from the LLM, the first response generated using the second prompt; and

send, the first response to the user device.

2 . The system of claim 1 , wherein the execution of the instructions further causes the system to:

receive a third prompt from the user device to be input to the LLM;

generate a second classification of the third prompt by at least using the first ML model;

select the third ML model based at least in part on the second classification;

provide a second input to the third ML model based at least in part on the third prompt and the second classification;

determine a fourth prompt that is output by the third ML model based at least in part on the second input; and

provide the fourth prompt to the LLM.

3 . The system of claim 1 , wherein the execution of the instructions further causes the system to:

receive, by the second ML model, the classification;

select, using the second ML model and the classification, the template associated with the classification from the set of templates;

generate, by the second ML model, information elements based at least on the first prompt, the information elements corresponding to fields defined by the template; and

generate the second prompt using the fields defined by the template and the information elements.

4 . The system of claim 1 , wherein the execution of the instructions further causes the system to:

transmit a query to a data store, the query including the classification, the data store storing the set of templates;

receive, by the second ML model, the template associated with the classification from the data store;

receive, by the second ML model, the first prompt; and

generate, by the second ML model, information elements based at least on the first prompt, the information elements corresponding to fields defined by the template, wherein the second prompt includes the information elements.

5 . A computer-implemented method, comprising:

receiving a first prompt from a user device to be input to a large language model (LLM);

generating, by at least using a first machine learning (ML) model, a classification of the first prompt;

selecting, based at least in part on the classification, a second ML model from a plurality of ML models, the second ML model configured to output first LLM prompts based at least in part on a set of templates, the plurality of ML models comprising a third ML model associated with a different classification and configured to output second LLM prompts independently of any template in the set of templates;

providing a first input to the second ML model based at least in part on the first prompt and the classification;

determining a second prompt that is output by the second ML model based at least in part on the first input and a template associated with the classification;

providing the second prompt to the LLM;

receiving a first response from the LLM, the first response generated using the second prompt; and

sending, the first response to the user device.

6 . The computer-implemented method of claim 5 ,

wherein the template is associated with the classification.

7 . The computer-implemented method of claim 5 , wherein determining the second prompt that is output by the second ML model further comprises:

receiving, by the second ML model, the classification;

selecting, using the second ML model and the classification, the template associated with the classification from the set of templates; and

generating, by the second ML model, information elements based at least on the first prompt, the information elements corresponding to fields defined by the template, wherein the second prompt includes the information elements.

8 . The computer-implemented method of claim 5 , wherein determining the second prompt that is output by the second ML model further comprises:

transmitting a query to a data store, the query including the classification, the data store storing the set of templates;

receiving, by the second ML model, the template associated with the classification from the data store;

receiving, by the second ML model, the first prompt; and

generating, by the second ML model, information elements based at least on the first prompt, the information elements corresponding to fields defined by the template; wherein the second prompt includes the information elements.

9 . The computer-implemented method of claim 5 , wherein the second ML model is trained according to a training procedure, the training procedure comprising:

receiving a third prompt;

generating, by using the second ML model based at least in part on the third prompt, a fourth prompt;

receiving an indication that one of: (i) the third prompt or (ii) the fourth prompt were selected to be used as input to the LLM; and

updating parameters of the second ML model based at least in part on the indication.

10 . The computer-implemented method of claim 5 , wherein the second ML model is trained according to a training procedure, the training procedure comprising:

receiving a third prompt;

generating, by using the second ML model based at least in part on the third prompt, a fourth prompt;

inputting the third prompt to the LLM;

inputting the fourth prompt to the LLM;

receiving a first output from the LLM corresponding to the third prompt;

receiving a second output from the LLM corresponding to the fourth prompt;

receiving an indication that one of: (i) the first output or (ii) the second output was of greater quality; and

updating parameters of the second ML model based at least in part on the indication.

11 . One or more non-transitory computer-readable storage media storing instructions that, upon execution executable by one or more processors of a system, cause the system to perform operations comprising:

receiving a first prompt from a user device to be input to a large language model (LLM);

generating, by at least using a first machine learning (ML) model, a classification of the first prompt;

selecting, based at least in part on the classification, a second ML model from a plurality of ML models, the second ML model configured to output first LLM prompts based at least in part on a set of templates, the plurality of ML models comprising a third ML model associated with a different classification and configured to output second LLM prompts independently of any template in the set of templates;

providing a first input to the second ML model based at least in part on the first prompt and the classification;

determining a second prompt that is output by the second ML model based at least in part on the first input and a template associated with the classification;

providing the second prompt to the LLM;

receiving a first response from the LLM, the first response generated using the second prompt; and

sending, the first response to the user device.

12 . The non-transitory computer-readable storage medium of claim 11 , wherein execution of the instructions further causes the system to perform training operations comprising:

receiving a third prompt;

generating, by using the second ML model based at least in part on the third prompt, a fourth prompt;

inputting the fourth prompt to the LLM;

receiving a second output from the LLM corresponding to the fourth prompt;

receiving an indication of a quality of the second output; and

updating parameters of the second ML model based at least in part on the indication.

13 . The non-transitory computer-readable storage medium of claim 12 , wherein the indication of the quality is determined by performing at least one of: determining a number of keywords used in the second output, determining a number of key phrases used in the second output, determining a length of the second output, determining an order of key words used in the second output, determining punctuation used in the second output, determining which keywords were used in the second output, determining which key phrases were used in the second output, determining an accuracy of the second prompt, or determining a performance metric of the second prompt.

14 . The non-transitory computer-readable storage medium of claim 12 , wherein receiving the indication further comprises:

receiving the indication from an analysis system configured to indicate which one of the third prompt or the fourth prompt has a higher prompt value metric.

15 . The non-transitory computer-readable storage medium of claim 12 , wherein the indication of the quality is determined by at least receiving user input indicating a label to be associated with the second output, the label indicating the quality.

16 . The non-transitory computer-readable storage medium of claim 11 , wherein execution of the instructions further causes the system to perform additional operations comprising:

wherein the template is associated with the classification.

17 . The non-transitory computer-readable storage medium of claim 11 , wherein the instructions are part of a plugin for program code of an application hosted by the system, and wherein the application is configured by the plugin to replace the first prompt with the second prompt as an input to the LLM.

18 . The non-transitory computer-readable storage medium of claim 17 , wherein the plugin further configures the application to present the second prompt at a user interface of the user device.

Assignments (1)

References Cited (37)