Granted Patent B1

US 12,374,326 · App. 18/141,051 · Granted Jul 29, 2025

Inventors: Alexandros Potamianos (Santa Monica, CA); Arijit Biswas (Dublin, CA); Bonan Zheng (Torrance, CA); Anushree Venkatesh (San Mateo, CA); Yohan Jo (Sunnyvale, CA); Vincent Auvray (Scotts Valley, CA); Nikolaos Malandrakis (San Jose, CA); Aaron Challenner (Melrose, MA); Xinyan Zhao (Seattle, WA); Angeliki Metallinou (Mountain View, CA); David A Jara (Normandy Park, WA); Jiahui Li (Sunnyvale, CA); Ying Shi (Bellevue, WA); Nikko Strom (Kirkland, WA); Veerdhawal Pande (Walpole, MA)

Assignee: Amazon Technologies, Inc.

G10L15/1815G10L15/22G10L2015/223

Abstract

Claims (94)

1. A computer-implemented method comprising:

receiving, from a device, first input audio data corresponding to first speech of a first user;

performing automatic speech recognition (ASR) processing using the first input audio data to determine first ASR output data comprising a first transcript of the first speech;

based on the first input audio data and the first ASR output data, determining the first speech is directed at a second user instead of the device;

determining information included in the first speech is usable to respond to speech directed to the device;

based on determining the information included in the first speech is usable to respond to the speech directed to the device, sending the first ASR output data to a storage;

after sending the first ASR output data to the storage, receiving, from the device, second input audio data corresponding to second speech;

performing ASR processing using the second input audio data to determine second ASR output data comprising a second transcript of the second speech;

based on the second ASR output data, determining the second speech is directed to the device;

based on determining the second speech is directed at the device, receiving the first ASR output data from the storage;

after receiving the first ASR output data from the storage, determining rewritten ASR output data corresponding to the second ASR output data updated to include at least one word from the first ASR output data;

performing natural language understanding (NLU) processing using the rewritten ASR output data to determine NLU output data comprising an intent corresponding to the second speech;

based on the NLU output data, determining second output data responsive to the second speech; and

causing presentation of the second output data.

2. The computer-implemented method of claim 1 , further comprising:

receiving first input image data corresponding to at least a first image representing the first user, wherein determining the first speech is directed at the second user instead of the device is further based on the first input image data; and

receiving second input image data corresponding to at least a second image representing the first user, wherein determining the second speech is directed at the device is further based on the second input image data.

3. The computer-implemented method of claim 1 , further comprising:

based on determining the second speech is directed at the device, receiving, from the storage, third ASR output data comprising a third transcript of third speech, wherein the third speech was received prior to the first speech, and the third ASR output data was stored based on the third speech being directed at a third user instead of the device; and

determining the second ASR output data is semantically similar to the first ASR output data instead of the third ASR output data, wherein the rewritten ASR output data includes the at least one word from the first ASR output data instead of the third ASR output data based on the second ASR output data being semantically similar to the first ASR output data instead of the third ASR output data.

4. The computer-implemented method of claim 1 , further comprising:

determining usage data including third ASR output data corresponding to third speech of a dialog;

determining the first speech is semantically similar to the third speech; and

sending the first ASR output data to the storage is further based on the first speech being semantically similar to the third speech.

5. A computer-implemented method comprising:

receiving, from a device, first input audio data corresponding to first speech of a first user;

determining the first speech is directed at the device;

based on determining the first speech is directed at the device, receiving, from a storage, a transcript of second speech, wherein the second speech was received prior to the first speech, and the transcript was stored based on the second speech being directed at a second user instead of the device;

after receiving the transcript from the storage, determining first natural language data corresponding to the first speech updated to include at least one word from the second speech;

performing natural language understanding (NLU) processing using the first natural language data to determine NLU output data comprising a first intent corresponding to at least one of the first speech and the second speech;

based on the NLU output data, determining first output data responsive to at least one of the first speech and the second speech; and

causing presentation of the first output data.

6. The computer-implemented method of claim 5 , further comprising:

prior to receiving the first input audio data, receiving second input audio data corresponding to the second speech;

determining the second speech is directed at the second user instead of the device;

determining information included in the second speech is usable to respond to speech directed to the device; and

based on determining the information included in the second speech is usable to respond to the speech directed to the device, sending the transcript of the second speech to the storage.

7. The computer-implemented method of claim 6 , further comprising:

determining an entity referenced in the second speech; and

determining the entity corresponds to an entity type capable of being processed using NLU processing, wherein determining the information included in the second speech is usable to respond to the speech directed to the device is based on the entity corresponding to the entity type.

8. The computer-implemented method of claim 6 , further comprising:

determining usage data including a second transcript of third speech; and

determining the second speech is semantically similar to the third speech, wherein sending the second speech to the storage is further based on the second speech being semantically similar to the third speech.

9. The computer-implemented method of claim 5 , further comprising:

receiving input image data corresponding to at least a first image representing the first user, wherein:

determining the first speech is directed at the device is further based on the input image data.

10. The computer-implemented method of claim 5 , further comprising:

based on determining the first speech is directed at the device, receiving, from the storage, a second transcript of third speech, wherein the third speech was received prior to the first speech, and the second transcript was stored based on the third speech being directed at a third user instead of the device; and

determining the first speech is semantically similar to the second speech, instead of the third speech, wherein the first natural language data includes the at least one word from the second speech, instead of the third speech based on the first speech being semantically similar to the second speech, instead of the third speech.

11. The computer-implemented method of claim 5 , further comprising:

receiving third input audio data corresponding to third speech;

determining a first portion of the third speech was spoken by the first user;

determining a second portion of the third speech was spoken by the second user;

determining the first portion of the third speech is directed at the device; and

determining the second portion of the third speech is directed at the first user.

12. The computer-implemented method of claim 5 , further comprising:

determining the second speech was spoken by the first user; and

based on the second speech being spoken by the first user, determining the first output data to include a name of the first user.

13. A computing system comprising:

at least one processor; and

at least one memory comprising instructions that, when executed by the at least one processor, cause the computing system to:

receive, from a device, first input audio data corresponding to first speech of a first user;

determine the first speech is directed at the device;

based on determining the first speech is directed at the device, receive, from a storage, a transcript of second speech, wherein the second speech was received prior to the first speech, and the transcript was stored based on the second speech being directed at a second user instead of the device;

after receiving the transcript from the storage, determine first natural language data corresponding to the first speech updated to include at least one word from the second speech;

perform natural language understanding (NLU) processing using the first natural language data to determine NLU output data comprising a first intent corresponding to at least one of the first speech and the second speech;

based on the NLU output data, determine first output data responsive to at least one of the first speech and the second speech; and

cause presentation of the first output data.

14. The computing system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

prior to receiving the first input audio data, receive second input audio data corresponding to the second speech;

determine the second speech is directed at the second user instead of the device;

determining information included in the second speech is usable to respond to speech directed to the device; and

based on determining the information included in the second speech is usable to respond to the speech directed to the device, send the transcript of the second speech to the storage.

15. The computing system of claim 14 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

determine an entity referenced in the second speech; and

determine the entity corresponds to an entity type capable of being processed using NLU processing, wherein determining the information included in the second speech is usable to respond to the speech directed to the device is based on the entity corresponding to the entity type.

16. The computing system of claim 14 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

determine usage data including a second transcript of third speech; and

determine the second speech is semantically similar to the third speech, wherein sending the second speech to the storage is further based on the second speech being semantically similar to the third speech.

17. The computing system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

receive input image data corresponding to at least a first image representing the first user, wherein:

determine the first speech is directed at the device is further based on the input image data.

18. The computing system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

based on determining the first speech is directed at the device, receive, from the storage, a second transcript of third speech, wherein the third speech was received prior to the first speech, and the second transcript was stored based on the third speech being directed at a third user instead of the device; and

determine the first speech is semantically similar to the second speech, instead of the third speech, wherein the first natural language data includes the at least one word from the second speech, instead of the third speech based on the first speech being semantically similar to the second speech, instead of the third speech.

19. The computing system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

receive third input audio data corresponding to third speech;

determine a first portion of the third speech was spoken by the first user;

determine a second portion of the third speech was spoken by the second user;

determine the first portion of the third speech is directed at the device; and

determine the second portion of the third speech is directed at the first user.

20. The computing system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

determine the second speech was spoken by the first user; and

based on the second speech being spoken by the first user, determine the first output data to include a name of the first user.

Assignments (1)

References Cited (34)

Cited By (1)

US 12,499,883