Granted Patent B2

US 12,424,210 · App. 18/456,949 · Granted Sep 23, 2025

Inventors: Xing Fan (Redmond, WA); Chenlei Guo (Redmond, WA); Narendra Gyanchandani (Sammamish, WA); Hyungseo Park (Seattle, WA)

Assignee: Amazon Technologies, Inc.

G10L15/183G10L15/22G10L2015/223G10L15/30

Abstract

Claims (139)

1. A computer-implemented method comprising:

receiving first audio data corresponding to a first spoken input;

performing automatic speech recognition (ASR) processing using the first audio data to generate a first transcript of the first spoken input;

receiving first potential response data from a first component configured to perform a first action associated with the first spoken input;

receiving second potential response data from a second component configured to perform a second action associated with the first spoken input;

receiving third potential response data from a third component configured to perform the second action;

generating a first prompt including the first potential response data, the second potential response data, the third potential response data, and the first transcript, wherein the first prompt is a first request for a language model to determine a first output responsive to the first spoken input;

processing, using the language model, the first prompt to generate first model output data indicating:

the first potential response data is responsive to the first spoken input, and

the second potential response data is responsive to the first spoken input;

based at least in part on the first model output data, causing the first component to perform the first action; and

based at least in part on the first model output data, causing the second component to perform the second action.

2. The computer-implemented method of claim 1 , further comprising:

receiving second audio data corresponding to a second spoken input;

performing ASR processing using the second audio data to generate a second transcript of the second spoken input;

receiving fourth potential response data from a fourth component, the fourth potential response data associated with the second spoken input;

receiving fifth potential response data from a fifth component, the fifth potential response data associated with the second spoken input;

generating a second prompt including the fourth potential response data, the fifth potential response data, and the second transcript, wherein the second prompt is a second request for the language model to determine a second output responsive to the first spoken input;

processing, using the language model, the second prompt to generate second model output data indicating:

the fourth potential response data is responsive to the second spoken input, and

the fifth potential response data is responsive to the second spoken input, wherein the second model output data includes second output data representing a summary of the fourth potential response data and the fifth potential response data; and

causing presentation of the second output data in response to the second spoken input.

3. The computer-implemented method of claim 1 , wherein the first potential response data is associated with a first task of the first spoken input, the second potential response data and the third potential response data are associated with a second task of the first spoken input, and the method further comprises:

prior to receiving the second potential response data and the third potential response data, receiving fourth potential response data from a fourth component configured to perform a fourth action associated with the first spoken input, wherein the fourth potential response data is associated with the first task;

generating a second prompt including the first potential response data, the fourth potential response data, and the first transcript, wherein the second prompt is a second request for the language model to determine a second output responsive to the first task of the first spoken input; and

processing, using the language model, the second prompt to generate second model output data indicating the first potential response data is responsive to the first task of the first spoken input, instead of the fourth potential response data.

4. The computer-implemented method of claim 1 , further comprising:

prior to generating the first prompt, generating a second prompt including the first potential response data, the second potential response data, the third potential response data, and the first transcript, wherein the second prompt is a second request for the language model to determine a second output responsive to the first spoken input;

processing, using the language model, the second prompt to generate second model output data including:

an indication of an ambiguity associated with the first spoken input, and

a third request for first data associated with the ambiguity;

based at least in part on the second model output data, sending, to a fourth component, the second model output data; and

receiving, from the fourth component, first context data associated with the ambiguity, wherein:

the first prompt further includes the first context data, and

the first model output data is unassociated with the ambiguity.

5. A computer-implemented method comprising:

receiving first input data;

receiving first potential response data from a first component configured to perform a first action associated with the first input data;

receiving second potential response data corresponding to a second component configured to perform a second action associated with the first input data;

determining a first prompt including the first potential response data, the second potential response data, the first input data, wherein the first prompt is a first request for a language model to determine a first output responsive to the first input data;

processing, using the language model, the first prompt to generate first model output data indicating the first action is responsive to the first input data; and

causing the first component to perform the first action.

6. The computer-implemented method of claim 5 , further comprising:

receiving second input data;

receiving third potential response data from a third component, the third potential response data associated with the second input data;

receiving fourth potential response data from a fourth component, the fourth potential response data associated with the second input data;

generating a second prompt including the third potential response data, the fourth potential response data, and the second input data, wherein the second prompt is a second request for the language model to determine a second output responsive to the second input data;

processing, using the language model, the second prompt to generate second model output data indicating:

the third potential response data is responsive to the second input data, and

the fourth potential response data is responsive to the second input data, wherein the second model output data includes second output data representing a summary of the third potential response data and the fourth potential response data; and

causing presentation of the second output data in response to the second input data.

7. The computer-implemented method of claim 5 , wherein the first potential response data is associated with a first task of the first input data, the second potential response data is associated with a second task of the first input data, and the method further comprises:

prior to receiving the second potential response data, receiving third potential response data corresponding to a third component configured to perform a third action associated with the first input data, wherein the third potential response data is associated with the first task;

determining a second prompt including the first potential response data and the third potential response data, wherein the second prompt is a second request for the language model to determine a second output responsive to the first task of the first input data; and

processing, using the language model, the second prompt to generate second model output data indicating the first potential response data is responsive to the first task of the first input data, instead of the third potential response data.

8. The computer-implemented method of claim 5 , further comprising:

prior to generating the first prompt, generating a second prompt including the first potential response data, the second potential response data, and the first input data, wherein the second prompt is a second request for the language model to determine a second output responsive to the first input data;

processing, using the language model, the second prompt to generate second model output data including:

an indication of an ambiguity associated with the first input data, and

a third request for first data associated with the ambiguity;

based at least in part on the second model output data, sending, to a third component, the second model output data; and

receiving, from the third component, first context data associated with the ambiguity, wherein:

the first prompt further includes the first context data, and

the first model output data is unassociated with the ambiguity.

9. The computer-implemented method of claim 5 , wherein the first action corresponds to a first device performing a third action, and the method further comprises:

determining first output data corresponding to the first action, the first output data indicating performance of the first action by the first device; and

after causing the first component to perform the first action, causing presentation of the first output data.

10. The computer-implemented method of claim 5 , wherein the first action corresponds to a first device performing a third action, and the method further comprises:

receiving third potential response data corresponding to a third component configured to perform a fourth action associated with the first input data, wherein:

the fourth action corresponds to a second device performing a fifth action,

the first prompt further includes the third potential response data, and

the first model output data further indicates that the fourth action is responsive to the first input data; and

causing the third component to perform the fourth action, wherein:

causing the third component to perform the fourth action includes causing the second device to perform the fifth action, and

causing the first component to perform the first action includes causing the first device to perform the third action.

11. The computer-implemented method of claim 5 , further comprising:

prior to generating the first prompt, generating a second prompt including the first potential response data, the second potential response data, and the first input data, wherein the second prompt is a second request for the language model to determine a second output responsive to the first input data;

processing, using the language model, the second prompt to generate second model output data including:

an indication of an ambiguity associated with the first input data, and

a third request for context data associated with the ambiguity;

based at least in part on the second model output data, determining second output data corresponding to the second model output data;

causing presentation of the second output data; and

receiving natural language data corresponding to a user input associated with the ambiguity, wherein:

the first prompt further includes the user input, and

the first model output data is unassociated with the ambiguity.

12. The computer-implemented method of claim 5 , wherein the first component is a skill and the second component is a large language model (LLM) based agent.

13. A computing system comprising:

at least one processor; and

at least one memory including instructions that, when executed by the at least one processor, cause the computing system to:

receive first input data;

receive first potential response data from a first component configured to perform a first action associated with the first input data;

receive second potential response data corresponding to a second component configured to perform a second action associated with the first input data;

determine a first prompt including the first potential response data, the second potential response data, the first input data, wherein the first prompt is a first request for a language model to determine a first output responsive to the first input data;

process, using the language model, the first prompt to generate first model output data indicating the first action is responsive to the first input data; and

cause the first component to perform the first action.

14. The computing system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

receive second input data;

receive third potential response data from a third component, the third potential response data associated with the second input data;

receive fourth potential response data from a fourth component, the fourth potential response data associated with the second input data;

generate a second prompt including the third potential response data, the fourth potential response data, and the second input data, wherein the second prompt is a second request for the language model to determine a second output responsive to the second input data;

process, using the language model, the second prompt to generate second model output data indicating:

the third potential response data is responsive to the second input data, and

the fourth potential response data is responsive to the second input data, wherein the second model output data includes second output data representing a summary of the third potential response data and the fourth potential response data; and

cause presentation of the second output data in response to the second input data.

15. The computing system of claim 13 , wherein the first potential response data is associated with a first task of the first input data, the second potential response data is associated with a second task of the first input data, and wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

prior to receiving the second potential response data, receive third potential response data corresponding to a third component configured to perform a third action associated with the first input data, wherein the third potential response data is associated with the first task;

determine a second prompt including the first potential response data and the third potential response data, wherein the second prompt is a second request for the language model to determine a second output responsive to the first task of the first input data; and

process, using the language model, the second prompt to generate second model output data indicating the first potential response data is responsive to the first task of the first input data, instead of the third potential response data.

16. The computing system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

prior to generating the first prompt, generate a second prompt including the first potential response data, the second potential response data, and the first input data, wherein the second prompt is a second request for the language model to determine a second output responsive to the first input data;

process, using the language model, the second prompt to generate second model output data including:

an indication of an ambiguity associated with the first input data, and

a third request for first data associated with the ambiguity;

based at least in part on the second model output data, send, to a third component, the second model output data; and

receive, from the third component, first context data associated with the ambiguity, wherein:

the first prompt further includes the first context data, and

the first model output data is unassociated with the ambiguity.

17. The computing system of claim 13 , wherein the first action corresponds to a first device performing a third action, and wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

determine first output data corresponding to the first action, the first output data indicating performance of the first action by the first device; and

after causing the first component to perform the first action, cause presentation of the first output data.

18. The computing system of claim 13 , wherein the first action corresponds to a first device performing a third action, and wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

receive third potential response data corresponding to a third component configured to perform a fourth action associated with the first input data, wherein:

the fourth action corresponds to a second device performing a fifth action,

the first prompt further includes the third potential response data, and

the first model output data further indicates that the fourth action is responsive to the first input data; and

cause the third component to perform the fourth action, wherein:

causing the third component to perform the fourth action includes causing the second device to perform the fifth action, and

causing the first component to perform the first action includes causing the first device to perform the third action.

19. The computing system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

prior to generating the first prompt, generate a second prompt including the first potential response data, the second potential response data, and the first input data, wherein the second prompt is a second request for the language model to determine a second output responsive to the first input data;

process, using the language model, the second prompt to generate second model output data including:

an indication of an ambiguity associated with the first input data, and

a third request for context data associated with the ambiguity;

based at least in part on the second model output data, determine second output data corresponding to the second model output data;

cause presentation of the second output data; and

receive natural language data corresponding to a user input associated with the ambiguity, wherein:

the first prompt further includes the user input, and

the first model output data is unassociated with the ambiguity.

20. The computing system of claim 13 , wherein the first component is a skill and the second component is a large language model (LLM) based agent.

Assignments (1)

Continuity (1)

References Cited (31)

Cited By (1)

US 12,718,010