Granted Patent B1

US 11,211,058 · App. 16/577,394 · Granted Dec 28, 2021

Inventors: Aaron Eakin (Seattle, WA); Angela Sun (Seattle, WA); Ankur Gandhe (Bothell, WA); Ariya Rastrow (Seattle, WA); Chenlei Guo (Redmond, WA); Xing Fan (Redmond, WA)

Assignee: Amazon Technologies, Inc.

G10L15/197G10L15/22G10L15/30G10L2015/223

Abstract

Claims (146)

1. A computer-implemented method comprising:

receiving, from a first device, first audio data representing a first utterance;

processing the first audio data using an automatic speech recognition (ASR) machine-learning model to determine ASR data comprising a first ASR hypothesis, a first score corresponding to the first ASR hypothesis, a second ASR hypothesis, and a second score corresponding to the second ASR hypothesis;

determining, based on the first score and the second score being below a threshold, an indication of uncertainty with respect to the processing of the first audio data;

determining that the indication of uncertainty relates to a first entity represented by the first ASR hypothesis and a second entity represented by the second ASR hypothesis;

generating output audio data requesting selection of the first entity or the second entity;

sending the output audio data to the first device;

receiving, from the first device in response to requesting selection of the first entity or the second entity, second audio data representing a second utterance;

performing speech processing on the second audio data to determine a selection of the first entity;

storing dialog session data comprising the first audio data, the indication of uncertainty, and the second audio data;

determining, using the dialog session data, that the indication of uncertainty relates to a user profile corresponding to the first device;

processing the dialog session data, the first audio data, and the second audio data to generate a user-specific ASR machine-learning model corresponding to the user profile; and

associating the user-specific ASR machine learning model with the user profile.

2. The computer-implemented method of claim 1 , further comprising:

receiving third audio data representing a third utterance;

determining that the third audio data corresponds to the user profile;

determining the user-specific ASR machine learning model corresponds to the user profile;

processing the third audio data using the ASR machine-learning model and the user-specific ASR machine learning model to determine a third ASR hypothesis and a third score, the third ASR hypothesis representing the first entity; and

generating output data using the third ASR hypothesis based at least in part on the third score satisfying a threshold.

3. The computer-implemented method of claim 1 , further comprising:

receiving, from the first device, third audio data representing a third utterance;

processing the third audio data using the ASR machine-learning model to determine a third ASR hypothesis and a third score, the third ASR hypothesis corresponding to a third entity;

determining a second indication of uncertainty based on the third score;

generating second output audio data requesting input with respect to the third entity;

sending the second output audio data to the first device;

receiving, from the first device, fourth audio data in response to the second output audio data;

storing second dialog session data comprising the third audio data, the second indication of uncertainty, and the fourth audio data;

determining that the second indication of uncertainty corresponds to a number of user profiles;

determining that the number of user profiles satisfies a condition; and

processing the second dialog session data and model data associated with the ASR machine-learning model to determine a second ASR machine-learning model.

4. The computer-implemented method of claim 1 , further comprising:

determining second dialog session data corresponding to a second user profile;

determining that the second dialog session data includes the indication of uncertainty relating to the first entity and the second entity;

determining a first location associated with the user profile;

determining a second location associated with the second user profile;

determining that the first location and the second location correspond to a geographic region;

associating the user-specific ASR machine learning model with the geographic region;

receiving third audio data representing a third utterance;

determining that the third audio data corresponds to a third user profile;

determining a third location associated with the third user profile;

determining that the third location corresponds to the geographic region;

determining that the user-specific ASR machine learning model is associated with the geographic region; and

processing the third audio data using the user-specific ASR machine learning model.

5. A computer-implemented method comprising:

receiving first data representing a first utterance;

determining an indication of ambiguity corresponding to the first data;

determining that the indication of ambiguity relates to second data represented in the first data, the second data representing an entity identified in the first data;

generating output audio data requesting input corresponding to the second data;

receiving, in response to requesting the input corresponding to the second data, input audio data representing a second utterance corresponding to the second data;

storing dialog data;

associating the first data, the indication of ambiguity, and third data corresponding to the input audio data with the dialog data;

determining, using the dialog data, that the indication of ambiguity relates to a user profile; and

processing the dialog data, the first data, and the third data to generate a user-specific machine-learning model corresponding to the user profile.

6. The computer-implemented method of claim 5 , further comprising:

determining confidence data corresponding to the input audio data;

determining that the confidence data is below a threshold;

processing the third data to determine a portion of the third data that represents to the entity; and

associating the portion of the third data with the dialog data.

7. The computer-implemented method of claim 5 , further comprising:

determining confidence data corresponding to the input audio data; and

determining that the confidence data is above a threshold,

wherein processing the dialog data to determine the user-specific machine-learning model comprises processing, in response to the confidence data being above the threshold, the dialog data to determine the user-specific machine-learning model.

8. The computer-implemented method of claim 5 , further comprising:

determining second dialog data corresponding to a second user profile;

determining that the second dialog data is associated with the indication of ambiguity;

determining a first location associated with the user profile;

determining a second location associated with the second user profile; determining that the first location and the second location correspond to a geographic region; and

associating the user-specific machine learning model with the geographic region.

9. The computer-implemented method of claim 5 , further comprising:

receiving second input audio data representing a third utterance;

determining that the second input audio data corresponds to the user profile;

determining the user-specific machine learning model corresponds to the user profile;

processing the second input audio data using the user-specific machine-learning model and an automatic speech recognition (ASR) model to determine fourth data and confidence data, the fourth data including a representation of the entity; and

generating output data using the fourth data based at least in part on the confidence data satisfying a threshold.

10. The computer-implemented method of claim 5 , further comprising:

receiving fourth data representing a third utterance;

receiving confidence data corresponding to the fourth data;

determining, using the confidence data, a second indication of ambiguity corresponding to the fourth data;

receiving second input audio data corresponding to the second indication of ambiguity;

storing second dialog data;

associating the fourth data, the second indication of ambiguity, and fifth data corresponding to the second input audio data with the dialog data;

determining that the second indication of ambiguity corresponds to a number of user profiles;

determining that the number of user profiles satisfies a condition;

determining an ASR machine-learning model used to determine the fourth data representing the third utterance; and

processing the second dialog data and model data corresponding to the ASR machine-learning model to determine a second ASR machine-learning model.

11. The computer-implemented method of claim 5 , further comprising:

determining a first entity and a second entity corresponding to the second data,

wherein generating the output audio data requesting the input corresponding to the second data comprises generating output audio data requesting selection of the first entity or the second entity, and

wherein receiving the input audio data comprises receiving the input audio data indicating selection of the first entity.

12. The computer-implemented method of claim 5 , further comprising:

processing the input audio data to determine text data;

determining an entity represented in the text data; and

generating output data using at least in part the entity, the output data corresponding to the first utterance.

13. A system comprising:

at least one processor; and

at least one memory including instructions that, when executed by the at least one processor, cause the system to:

receive first data representing a first utterance;

determine an indication of ambiguity corresponding to the first data;

determine that the indication of ambiguity relates to second data represented in the first data, the second data representing an entity identified in the first data;

generate output audio data requesting input corresponding to the second data;

receive, in response to requesting the input corresponding to the second data, input audio data representing a second utterance corresponding to the second data;

store dialog data;

associate the first data, the indication of ambiguity, and third data corresponding to the input audio data with the dialog data;

determine, using the dialog data, that the indication of ambiguity relates to a user profile; and

process the dialog data, the first data, and the third data to generate a user-specific machine-learning model corresponding to the user profile.

14. The system of claim 13 , wherein the instructions that, when executed by the at least one processor, further cause the system to:

determine confidence data corresponding to the input audio data;

determine that the confidence data is below a threshold;

process the third data to determine a portion of the third data that represents the entity; and

associate the portion of the third data with the dialog data.

15. The system of claim 13 , wherein the instructions that, when executed by the at least one processor, cause the system to receive the data further causes the system to:

determine confidence data corresponding to the input audio data; and

determine that the confidence data is above a threshold,

wherein the instructions to process the dialog data to determine the user-specific machine-learning model causes the system to process, in response to the confidence data being above the threshold, the dialog data to determine the user-specific machine-learning model.

16. The system of claim 13 , wherein the instructions that, when executed by the at least one processor, cause the system to receive the data further causes the system to:

determine second dialog data corresponding to a second user profile;

determine that the second dialog data is associated with the indication of ambiguity;

determine a first location associated with the user profile;

determine a second location associated with the second user profile;

determine that the first location and the second location correspond to a geographic region; and

associate the user-specific machine learning model with the geographic region.

17. The system of claim 13 , wherein the instructions that, when executed by the at least one processor, further cause the system to:

receive second input audio data representing a third utterance;

determine that the second input audio data corresponds to the user profile;

determine the user-specific machine learning model corresponds to the user profile;

process the second input audio data using the user-specific machine-learning model and an automatic speech recognition (ASR) model to determine fourth data and confidence data, the fourth data including a representation of the entity; and

generate output data using the fourth data based at least in part on the confidence data satisfying a threshold.

18. The system of claim 13 , wherein the instructions that, when executed by the at least one processor, further cause the system to:

receive fourth data representing a third utterance;

receive confidence data corresponding to the fourth data;

determine, using the confidence data, a second indication of ambiguity corresponding to the fourth data;

receive second input audio data corresponding to the second indication of ambiguity;

store second dialog data;

associate the fourth data, the second indication of ambiguity and fifth data corresponding to the second input audio data with the dialog data;

determine that the second indication of ambiguity corresponds to a number of user profiles;

determine that the number of user profiles satisfies a condition;

determine an ASR machine-learning model used to determine the fourth data representing the third utterance; and

process the second dialog data and model data corresponding to the ASR machine-learning model to determine a second ASR machine-learning model.

19. The system of claim 13 , wherein the instructions that, when executed by the at least one processor, further cause the system to:

determine a first entity and a second entity corresponding to the second data,

wherein the instructions to generate the output audio data requesting the input corresponding to the second data causes the system to generate output audio data requesting selection of the first entity or the second entity, and

wherein the instructions to receive the input audio data causes the system to receive the input audio data indicating selection of the first entity.

20. The system of claim 13 , wherein the instructions that, when executed by the at least one processor, further cause the system to:

process the input audio data to determine text data;

determine an entity represented in the text data; and

generate output data using at least in part the entity, the output data corresponding to the first utterance.

Assignments (1)

Cited By (15)

US 12,211,500 US 12,301,635 US 12,333,404 US 12,386,434 US 12,403,924 US 12,417,758 US 12,423,917 US 12,451,118 US 12,477,470 US 12,488,798 US 12,592,899 US 12,608,171 US 12,633,290 US 12,718,013 US 12,748,568