Granted Patent B1

US 11,081,104 · App. 15/838,917 · Granted Aug 3, 2021

Inventors: Chengwei Su (Belmont, MA); Sankaranarayanan Ananthakrishnan (Belmont, MA); Spyridon Matsoukas (Hopkinton, MA); Shirin Saleem (Belmont, MA); Rahul Gupta (Cambridge, MA); Kavya Ravikumar (Mercer Island, WA); John Will Crimmins (Seattle, WA); Kelly James Vanee (Shoreline, WA); John Pelak (Harvard, MA); Melanie Chie Bomke Gens (Seattle, WA)

Assignee: Amazon Technologies, Inc.

G10L15/063G06F16/313G06F40/295G06K9/00456G10L15/183G10L15/1815G10L15/22G10L15/26G10L15/32H04L67/306H04W4/02

Abstract

Claims (136)

1. A computer-implemented method comprising:

receiving, from a first device, input audio data corresponding to an utterance;

performing speech recognition processing on the input audio data to generate text data representing the utterance;

determining a natural language hypothesis representing a potential interpretation of the text data, wherein determining the natural language hypothesis comprises:

processing the text data using an intent classifier to generate an intent score representing a likelihood that an intent corresponds to the text data,

processing the text data using a named entity recognition (NER) model to identify a portion of the text data potentially relevant to the intent,

processing the text data using the NER model to generate a NER score indicating a likelihood the text data is relevant to the intent, and

processing the text data using a domain classification model to generate a domain score representing a likelihood that the text data corresponds to a domain;

determining a device type of the first device;

determining a feature vector comprising the intent score, the NER score and the domain score, wherein:

the intent score is located at a first position in the feature vector, the first position corresponding to at least the device type,

the NER score is located at a second position in the feature vector, the second position corresponding to at least the device type, and

the domain score is located at a third position in the feature vector, the third position corresponding to at least the device type;

sending the feature vector to a log linear model component comprising a weight vector, the weight vector comprising a plurality of weights, wherein the plurality of weights comprises:

a first weight corresponding to the first position,

a second weight corresponding to the second position, and

a third weight corresponding to the third position;

multiplying the intent score by the first weight to determine a weighted intent score;

multiplying the NER score by the second weight to determine a weighted NER score;

multiplying the domain score by the third weight to determine a weighted domain score;

determining a new score for the natural language hypothesis using at least the weighted intent score, the weighted NER score and the weighted domain score; and

causing a command corresponding to the intent to be executed.

2. The computer-implemented method of claim 1 , further comprising:

receiving, from the first device, a device identifier;

identifying, in a user profile, the device type using the device identifier;

determining the first position, the second position, and the third position using the device type;

populating the feature vector with the intent score in the first position, the NER score in the second position and the domain score in the third position;

determining a plurality of other positions in the feature vector corresponding to other device types; and

populating each of the plurality of other positions with a value of zero.

3. The computer-implemented method of claim 1 , further comprising:

determining context data corresponding to the utterance, the context data including data corresponding to hardware components corresponding to a device type;

determining the first device is associated with a second device corresponding to a different device type;

determining the second device is displaying video during the utterance;

determining a context profile corresponding to the displaying of video and the device type;

determining that the first position, the second position, and the third position correspond to the context profile;

determining a fourth position in the feature vector corresponding to data second context profile; and

populating the fourth position with a value of zero.

4. A system comprising:

at least one processor; and

at least one memory including instructions that, when executed by the at least one processor, cause the system to:

receive, from a device, a device identifier of the device and input data corresponding to a user input;

process text data representing the input data to determine a first plurality of scores for a first natural language hypothesis corresponding to the input data;

determine, based on at least the device identifier, that the input data corresponds to a first context profile;

populate a first feature vector with the first plurality of scores, the first plurality of scores being at a first plurality of positions in the first feature vector, the first plurality of positions corresponding to the first context profile, wherein a second plurality of positions in the first feature vector corresponds to a second context profile;

multiply the first feature vector by a weight vector including a first plurality of weights associated with the first context profile and a second plurality of weights associated with a second context profile to determine first result data;

determine a first new score for the first natural language hypothesis using the first result data; and

select the first natural language hypothesis for further operations based at least in part on the first new score.

5. The system of claim 4 , the memory further comprising instructions that, when executed by the at least one processor, further cause the system to:

process the text data using a first model to determine an intent score representing a likelihood that an intent corresponds to the text data;

process the text data using a second model to determine a named entity recognition (NER) score indicating a likelihood the text data is relevant to the intent;

process the text data using a third model to determine a domain score representing a likelihood that the text data corresponds to a domain; and

include in the first plurality of scores the intent score, the NER score and the domain score.

6. The system of claim 4 , the memory further comprising instructions that, when executed by the at least one processor, further cause the system to:

determine a second plurality of scores for a second natural language hypothesis corresponding to the input data;

populate a second feature vector using the second plurality of scores and the first context profile;

multiply the second feature vector by the first plurality of weights to determine third result data;

determine a second new score for the second natural language hypothesis using the third result data; and

determine the first new score is higher than the second new score.

7. The system of claim 4 , wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

determine a first subset of the first feature vector corresponding to the first context profile; and

populate the first subset of the first feature vector with the first plurality of scores.

8. The system of claim 7 , wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

determine a plurality of other subsets in the first feature vector corresponding to other context profiles; and

populate each of the plurality of other subsets with a value of zero.

9. The system of claim 7 , wherein:

the first plurality of weights is trained using a plurality of training utterances corresponding to the first context profile.

10. The system of claim 4 , wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

determining the first context profile corresponds to at least one processing rule; and

select the first natural language hypothesis further using the at least one processing rule.

11. The system of claim 4 , wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

receive, from an application component, first data indicating media content being sent from the application component to the device during a time the system receives the input data, wherein determining that the input data corresponds to the first context profile is additionally based on the first data.

12. The system of claim 4 , wherein the first plurality of weights are determined during a first training operation using utterance training data corresponding to a first context profile and the second plurality of weights are determined during a second training operation using utterance training data corresponding to a second context profile.

13. The system of claim 4 , wherein:

the input data is audio data corresponding to an utterance; and

the memory further comprises instructions, that when executed by the at least one processor, further cause the system to:

perform speech recognition processing on the audio data to generate the text data, and

determine the first natural language hypothesis based on the text data.

14. The system of claim 4 , wherein the first context profile comprises data representing one or more of a device type, a hardware capability, or a device state, and determining that the input data corresponds to the first context profile is additionally based on the device corresponding to the one or more of the device type, the hardware capability, or the device state.

15. The system of claim 4 , wherein the user input is an utterance and the input data includes audio data.

16. The system of claim 4 , wherein:

the first plurality of weights are included in a weight vector; and

the weight vector was determined by:

processing probability data corresponding to a training natural language hypothesis to generate modified probability data, the training natural language hypothesis corresponding to the first context profile;

processing the modified probability data to generate the first plurality of weights, the first plurality of weights comprising at least a first weight and a second weight; and

generating the weight vector to include the first weight at a first position and the second weight at a second position, the first position and the second position corresponding to the first context profile.

17. The system of claim 4 , wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

populate the weight vector by locating the first plurality of weights at a third plurality of positions in the weight vector corresponding to the first context profile and locating the second plurality of weights at a fourth plurality of positions in the weight vector corresponding to the second context profile.

18. The system of claim 17 , wherein:

determining the first result data includes determining a dot product of the first feature vector and the weight vector.

19. A computer-implemented method comprising:

receiving, from a device, a device identifier of the device and input data corresponding to a user input;

processing text data representing the input data to determine a first plurality of scores for a first natural language hypothesis corresponding to the input data;

determining, based on at least the device identifier, that the input data corresponds to a first context profile;

populating a first feature vector with the first plurality of scores, the first plurality of scores being at a first plurality of positions in the first feature vector, the first plurality of positions corresponding to the first context profile, wherein a second plurality of positions in the first feature vector corresponds to a second context profile;

multiplying the first feature vector by a weight vector including a first plurality of weights associated with the first context profile and a second plurality of weights associated with a second context profile to determine first result data;

determining a first new score for the first natural language hypothesis using the first result data; and

selecting the first natural language hypothesis for further operations based at least in part on the first new score.

20. The computer-implemented method of claim 19 , further comprising:

processing the text data using a first model to determine an intent score representing a likelihood that an intent corresponds to the text data;

processing the text data using a second model to determine a named entity recognition (NER) score indicating a likelihood the text data is relevant to the intent;

processing the text data using a third model to determine a domain score representing a likelihood that the text data corresponds to a domain; and

including in the first plurality of scores the intent score, the NER score and the domain score.

21. The computer-implemented method of claim 19 , further comprising:

determining a second plurality of scores for a second natural language hypothesis corresponding to the input data;

populating a second feature vector using the second plurality of scores and the first context profile;

multiplying the second feature vector by the first plurality of weights to determine third result data;

determining a second new score for the second natural language hypothesis using the third result data; and

determining the first new score is higher than the second new score.

22. The computer-implemented method of claim 19 , further comprising:

determining a first subset of the first feature vector corresponding to the first context profile; and

populating first subset of the first feature vector with the first plurality of scores.

23. The computer-implemented method of claim 22 , further comprising:

determining a plurality of other subsets in the first feature vector corresponding to other context profiles; and

populating each of the plurality of other subsets with a value of zero.

24. The computer-implemented method of claim 22 , wherein:

the first plurality of weights is trained using a plurality of training utterances corresponding to the first context profile.

25. The computer-implemented method of claim 19 , further comprising:

receiving, from an application component, first data indicating media content being sent from the application component to the device during a time the input data was received, wherein determining that the input data corresponds to the first context profile is additionally based on the first data.

26. The computer-implemented method of claim 19 , wherein the first plurality of weights are determined during a first training operation using utterance training data corresponding to a first context profile and the second plurality of weights are determined during a second training operation using utterance training data corresponding to a second context profile.

27. The computer-implemented method of claim 19 , wherein:

the input data is audio data corresponding to an utterance; and

the computer-implemented method further comprises:

performing speech recognition processing on the audio data to generate the text data, and

determining the first natural language hypothesis based on the text data.

28. The computer-implemented method of claim 19 , wherein the first context profile comprises data representing one or more of a device type, a hardware capability, or a device state, and determining that the input data corresponds to the first context profile is additionally based on the device corresponding to the one or more of the device type, the hardware capability, or the device state.

29. The computer-implemented method of claim 19 , wherein the user input is an utterance and the input data includes audio data.

30. The computer-implemented method of claim 19 , wherein:

the first plurality of weights are included in a weight vector; and

the weight vector was determined by:

processing probability data corresponding to a training natural language hypothesis to generate modified probability data, the training natural language hypothesis corresponding to the first context profile;

processing the modified probability data to generate the first plurality of weights, the first plurality of weights comprising at least a first weight and a second weight; and

generating the weight vector to include the first weight at a first position and the second weight at a second position, the first position and the second position corresponding to the first context profile.

31. The computer-implemented method of claim 19 , further comprising:

populating the weight vector by locating the first plurality of weights at a third plurality of positions in the weight vector corresponding to the first context profile and locating the second plurality of weights at a fourth plurality of positions in the weight vector corresponding to the second context profile.

32. The computer-implemented method of claim 31 , wherein:

determining the first result data includes determining a dot product of the first feature vector and the weight vector.

Assignments (1)

Continuity (1)

Cited By (11)

US 12,243,176 US 12,283,267 US 12,334,057 US 12,340,792 US 12,353,463 US 12,367,862 US 12,475,881 US 12,488,186 US 12,562,152 US 12,573,405 US 12,670,911