Natural language grammar improvement
A method for configuring natural language grammars is provided to include identifying a first transcription having a first automatic speech recognition (ASR) score and a first natural language understanding (NLU) score and identifying a second transcription having a second ASR score and a second NLU score. The method includes detecting that a difference between the first and second ASR scores has a signed value with an opposite sign than a sign of a signed value of a difference between the first and second NLU scores, and responsive to detecting the opposite sign providing, to an evaluator, the audio query and the first and second transcriptions, receiving, from the evaluator, an indication of which of the first and second transcriptions is a correct transcription, and adjusting a value implemented to calculate the first NLU score or a value implemented to calculate the second NLU score.
1. A computer-implemented method for configuring natural language grammars, the method comprising:
for a stored audio query,
generating, using a processor, a first transcription of the audio query, the first transcription having (i) first automatic speech recognition (ASR) score and (ii) a first natural language understanding (NLU) score, the first NLU score resulting from a first weighted grammar associated with the first transcription; and
identifying a second transcription of the audio query, the second transcription having (i) a second ASR score and (ii) a second NLU score;
detecting that a difference between the first ASR score and the second ASR score has signed value with an opposite sign than a sign of a signed value of a difference between the first NLU score and the second NLU score; and
responsive to detecting the opposite sign:
providing, to an evaluator, the audio query and the first and second transcriptions;
receiving, from the evaluator, an indication of which of the first transcription and the second transcription is a correct transcription;
adjusting, using a processor, a value implemented to calculate the first NLU score or a value implemented to calculate the second NLU score; and
calculating at least one of the first NLU score and the second NLU score using the adjusted value.
2. The method of claim 1 ,
wherein the second NLU score results from a second weighted grammar associated with the second transcription; and
wherein the second weighted grammar is different from the first weighted grammar.
3. The method of claim 2 , wherein the adjusting comprises:
increasing a weight for the weighted grammar associated with the correct transcription; or
decreasing a weight for the weighted grammar associated with the transcription, of the first and second transcriptions, that is not identified as the correct transcription.
4. The method of claim 3 , wherein an amount of increasing the weight or decreasing the weight is selected, such that the amount will cause the signed value of the difference between the first NLU score and the second NLU score to have the same sign as the signed value of the difference between the first ASR score and the second ASR score.
5. The method of claim 2 , wherein the first weighted grammar is from a domain that is associated with the first transcription and the second weighted grammar is from a domain that is associated with a second transcription.
6. The method of claim h further comprising:
performing a regression diagnostic of a set of test audio queries using weighted grammars with adjusted weights to identify test audio queries that have inconsistent interpretations using the adjusted weights.
7. The method of claim 1 , further comprising:
determining whether or not the ASR score of the correct transcription is lower than the ASR score of the transcription that is not identified as the correct transcription; and
when the ASR score of the correct transcription is determined to be lower, providing an indication of a domain error, meaning that an incorrect domain is associated with the correct transcription.
8. The method of claim 7 , further including providing, to a developer, the correct transcription.
9. The method of claim 8 , further including highlighting or identifying words that are present within the correct transcription and that are not present within the transcription that is not identified as the correct transcription.
10. The method of claim 1 , wherein the evaluator is a human.
11. The method of claim 1 , wherein the evaluator is a computer system that employs artificial intelligence to provide the indication of which transcription of the first and second transcription is the correct transcription.
12. The method of claim 1 , wherein the evaluator identifies the correct transcription by comparing the audio query to the first transcription and the second transcription to determine which of the first transcription and the second transcription more closely matches the audio query.
13. The method of claim 1 , further comprising:
identifying one or more domains that support at least one of the first and second transcriptions;
calculating a weight for each identified domain, the calculated weight for each respective domain indicating a likelihood that the at least one of the first and second transcriptions supported by the respective domain is the most accurate transcription in comparison to other identified domains; and
if a calculated weight for a domain identified as supporting the correct transcription is less than a calculated weight for a domain identified as supporting the transcription that is not identified as the correct transcription, then (i) determining that accuracy can be improved by adjusting weights of the domains and (ii) automatically adjusting the weights of the domains according to raw counts of the correct transcription within each identified domain.