Granted Patent B1

US 12,657,393 · App. 18/678,460 · Granted Jun 16, 2026

Inventors: Nimisha Nitin Mishra (Seattle, WA); Sri Harsha Reddy Wdaru (Seattle, WA); Shiv Ranjini Rajagopal (Mountain View, CA); Krishna Kotnana (Sammamish, WA); Xiangxing Lu (Bellevue, WA); Ismail Enchikalathil Jelal (Seattle, WA); Jie Liang (Bothell, WA); Elliott George Olson (Bellevue, WA)

Assignee: Amazon Technologies, Inc.

G06F40/30G10L15/18G10L15/22G10L15/26

Abstract

Claims (114)

1 . A computer-implemented method comprising:

receiving a first set of natural language input data corresponding to a first set of user inputs, the first set of natural language input data to be processed using an application;

based on the first set of natural language input data, retrieving, from a first optimization cache associated with a first optimization component, a first set of historical output data including a first set of prior responses, the first optimization component being associated with a speculative processing technique;

based on the first set of natural language input data, retrieving, from a second optimization cache associated with a second optimization component, a second set of historical output data including a second set of prior responses;

determining the first set of historical output data is larger than the second set of historical output data;

based on the first set of historical output data being larger than the second set of historical output data, determining first optimization plan data indicating the first optimization component is to be used to generate responses to input data corresponding to the application;

determining an application identifier (ID) associated with the application;

storing the first optimization plan data in a first storage in association with the application ID;

receiving first natural language input data corresponding to a first user input, the first natural language input data to be processed using the application;

receiving first context data including the application ID associated with the application;

using the application ID, retrieving, from the first storage, the first optimization plan data;

based on the first optimization plan data, sending the first natural language input data and the first context data to the first optimization component;

retrieving, from the first optimization cache, first historical output data corresponding to a first prior response to a first prior user input, the first prior user input determined by the first optimization component to correspond to the first user input;

sending, to the application, an indication of the first optimization component to instruct processing by a first language model of the first natural language input data; and

sending the first historical output data to the application to be processed by the first language model corresponding to the application to generate first response data corresponding to the first user input, wherein the first response data includes at least a first portion of the first historical output data.

2 . The computer-implemented method of claim 1 , further comprising:

receiving, from the application, feedback data indicating user satisfaction with the first response data;

based on the first response data and the feedback data, determining performance metric data corresponding to the processing of the first optimization component; and

storing the first response data in the first optimization cache further based on the performance metric data.

3 . The computer-implemented method of claim 1 , wherein the first optimization plan data further indicates the second optimization component is to be used to generate responses to input data corresponding to the application, and the method further comprises:

based on the first optimization plan data, sending the first natural language input data and the first context data to the second optimization component;

retrieving, from the second optimization cache associated with the second optimization component, second historical output data corresponding to a second prior response to a second prior user input, the second prior user input determined by the second optimization component to correspond to the first user input;

determining a first semantic similarity between the first natural language input data and the first historical output data;

determining a second semantic similarity between the first natural language input data and the second historical output data; and

based on the first semantic similarity and the second semantic similarity, determining to send the first historical output data to the application, instead of the second historical output data.

4 . The computer-implemented method of claim 1 , further comprising:

based on the first response data, determining optimization metrics data representing a performance of processing performed by the first optimization component with respect to the first natural language input data; and

based on the optimization metrics data, determining second optimization plan data indicating that the second optimization component is to be used to generate responses to input data corresponding to the application, instead of the first optimization component.

5 . A computer-implemented method comprising:

receiving a first set of historical input data;

based on the first set of historical input data, determining a first optimization component is to process future input data associated with the first set of historical input data, wherein the first optimization component is associated with a first technique to be performed to respond to the future input data;

receiving first input data to be processed, the first input data being associated with the first set of historical input data;

based on the first input data being associated with the first set of historical input data, selecting the first optimization component to process the first input data;

determining, by the first optimization component, first historical output data corresponding to a first prior response to a first prior input, wherein the first prior input corresponds to the first input data;

sending an indication of the first optimization component to cause a first language model to process the first input data according to the first technique; and

sending the first historical output data to be processed by the first language model to generate first output data responsive to the first input data.

6 . The computer-implemented method of claim 5 , wherein the first historical output data is retrieved from a first optimization cache associated with the first optimization component, and the method further comprises:

receiving feedback data indicating user satisfaction with the first output data;

based on the first output data and the feedback data, determining performance metric data associated with the processing of the first optimization component; and

storing the first output data in the first optimization cache based on the performance metric data.

7 . The computer-implemented method of claim 5 , wherein the first input data is to be processed using an application, and the method further comprises:

receiving first context data including an application identifier (ID) associated with the application;

using the application ID, retrieving, from a first storage, optimization plan data indicating the first optimization component and a second optimization component of a plurality of optimization components is to be used to generate responses to input data corresponding to the application;

determining, from a second optimization cache associated with the second optimization component, second historical output data corresponding to a second prior response to a second prior input, wherein the second prior input corresponds to the first input data;

determining a first semantic similarity between the first input data and the first historical output data;

determining a second semantic similarity between the first input data and the second historical output data; and

based on the first semantic similarity and the second semantic similarity, determining to send the first historical output data to the application, instead of the second historical output data.

8 . The computer-implemented method of claim 5 , wherein determining the first optimization component is to process the future input data associated with the first set of historical input data further comprises:

based on the first set of historical input data, retrieving, from a first optimization cache associated with the first optimization component, a first set of historical output data determined to be similar to the first set of historical input data;

based on the first set of historical input data, retrieving, from a second optimization cache associated with a second optimization component, a second set of historical output data determined to be similar to the first set of historical input data;

determining the first set of historical output data is larger than the second set of historical output data;

based on the first set of historical output data being larger than the second set of historical output data, determining optimization plan data indicating the first optimization component is preferred to be used to generate responses to the future input data over the second optimization component; and

storing the optimization plan data in a first storage, wherein selecting the first optimization component to process the first input data is based at least in part on the optimization plan data.

9 . The computer-implemented method of claim 5 , further comprising:

determining optimization metrics data representing a performance of processing performed by the first optimization component and a second optimization component with respect to a first set of inputs;

based on the optimization metrics data, determining a first score representing a performance of processing performed by the first optimization component with respect to the first set of inputs;

based on the optimization metrics data, determining a second score representing a performance of processing performed by the second optimization component with respect to the first set of inputs; and

based on the first score and the second score, determining optimization plan data indicating that the first optimization component is to be used to generate responses to the future input data, instead of the second optimization component.

10 . The computer-implemented method of claim 5 , further comprising:

processing the first output data to determine that the first output data is associated with processing performed by a second optimization component; and

storing the first output data in a second optimization cache associated with the second optimization component.

11 . The computer-implemented method of claim 5 , wherein the first input data is to be processed by an application, and the method further comprises:

receiving first context data including an application identifier (ID) associated with the application;

using the application ID, retrieving, from a first storage, optimization plan data indicating the first optimization component and a second optimization component of a plurality of optimization components is to be used to generate responses to input data corresponding to the application, the second optimization component being a preferred optimization component;

performing a first query of a second optimization cache associated with the second optimization component, using the first input data, for second historical output data corresponding to a second prior response to a second prior input, wherein the second prior input corresponds to the first input data;

determining a cache miss occurs with respect to the first query; and

based on the cache miss, determining the first historical output data.

12 . The computer-implemented method of claim 5 , wherein:

the first historical output data includes first model output data corresponding to a first task and second model output data corresponding to a second task, and

the first historical output data enables the first language model to generate third model output data corresponding to the first task and fourth model output data corresponding to the second task at least partially in parallel to generate the first output data.

13 . A computing system comprising:

at least one processor; and

at least one memory comprising instructions that, when executed by the at least one processor, cause the computing system to:

receive a first set of historical input data;

based on the first set of historical input data, determine a first optimization component is to process future input data associated with the first set of historical input data, wherein the first optimization component is associated with a first technique to be performed to respond to the future input data;

receive first input data to be processed, the first input data being associated with the first set of historical input data;

based on the first input data being associated with the first set of historical input data, select the first optimization component to process the first input data;

determine, by the first optimization component, first historical output data corresponding to a first prior response to a first prior input, wherein the first prior input corresponds to the first input data;

send an indication of the first optimization component to cause a first language model to process the first input data according to the first technique; and

send the first historical output data to be processed by the first language model to generate first output data responsive to the first input data.

14 . The computing system of claim 13 , wherein the first historical output data is retrieved from a first optimization cache associated with the first optimization component, and wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

receive feedback data indicating user satisfaction with the first output data;

based on the first output data and the feedback data, determine performance metric data associated with the processing of the first optimization component; and

store the first output data in the first optimization cache further based on the performance metric data.

15 . The computing system of claim 13 , wherein the first input data is to be processed using an application and wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

receive first context data including an application identifier (ID) associated with the application;

use the application ID, retrieving, from a first storage, optimization plan data indicating the first optimization component and a second optimization component of a plurality of optimization components is to be used to generate responses to input data corresponding to the application;

determine, from a second optimization cache associated with the second optimization component, second historical output data corresponding to a second prior response to a second prior input, wherein the second prior input corresponds to the first input data;

determine a first semantic similarity between the first input data and the first historical output data;

determine a second semantic similarity between the first input data and the second historical output data; and

based on the first semantic similarity and the second semantic similarity, determine to send the first historical output data to the application, instead of the second historical output data.

16 . The computing system of claim 13 , wherein the instructions that cause the computing system to determine the first optimization component is to process the future input data associated with the first set of historical input data comprise further instructions that, when executed by the at least one processor, further cause the computing system to:

based on the first set of historical input data, retrieve, from a first optimization cache associated with the first optimization component, a first set of historical output data determined to be similar to the first set of historical input data;

based on the first set of historical input data, retrieve, from a second optimization cache associated with a second optimization component, a second set of historical output data determined to be similar to the first set of historical input data;

determine the first set of historical output data is larger than the second set of historical output data;

based on the first set of historical output data being larger than the second set of historical output data, determine optimization plan data indicating the first optimization component is preferred to be used to generate responses to the future input data over the second optimization component; and

store the optimization plan data in a first storage, wherein selecting the first optimization component to process the first input data is based at least in part on the optimization plan data.

17 . The computing system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

determine optimization metrics data representing a performance of processing performed by the first optimization component and a second optimization component with respect to a first set of inputs;

based on the optimization metrics data, determine a first score representing a performance of processing performed by the first optimization component with respect to the first set of inputs;

based on the optimization metrics data, determine a second score representing a performance of processing performed by the second optimization component with respect to the first set of inputs; and

based on the first score and the second score, determine optimization plan data indicating that the first optimization component is to be used to generate responses to the future input data, instead of the second optimization component.

18 . The computing system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

process the first output data to determine that the first output data is associated with processing performed by a second optimization component; and

store the first output data in a second optimization cache associated with the second optimization component.

19 . The computing system of claim 13 , wherein the first input data is to be processed by an application and wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

receive first context data including an application identifier (ID) associated with the application;

using the application ID, retrieve, from a first storage, optimization plan data indicating the first optimization component and a second optimization component of a plurality of optimization components is to be used to generate responses to input data corresponding to the application, the second optimization component being a preferred optimization component;

perform a first query of a second optimization cache associated with the second optimization component, using the first input data, for second historical output data corresponding to a second prior response to a second prior input, wherein the second prior input corresponds to the first input data;

determine a cache miss occurs with respect to the first query; and

based on the cache miss, determine the first historical output data.

20 . The computing system of claim 13 , wherein:

the first historical output data includes first model output data corresponding to a first task and second model output data corresponding to a second task, and

the first historical output data enables the first language model to generate third model output data corresponding to the first task and fourth model output data corresponding to the second task at least partially in parallel to generate the first output data.

Assignments (1)

References Cited (14)