Granted Patent B1

US 10,789,943 · App. 16/120,182 · Granted Sep 29, 2020

Inventors: Larissa Lapshina (Shirley, MA); Mahnoosh Mehrabani Sharifbad (North Bethesda, MD); David Thomson (North Salt Lake, UT); Yoryos Yeracaris (Boston, MA)

Assignee: Interactions LLC

G10L15/1815G10L15/22G10L15/30

Abstract

Claims (98)

1. A computer-implemented method comprising:

obtaining, for a given sub-domain encompassed by a given domain, an input data set comprising a plurality of intent recognition automation rates and intent recognition accuracies corresponding to the automation rates;

obtaining, for each of a plurality of sub-domains disjoint from the given sub-domain and disjoint from each other and encompassed by the given domain, an input data set for a curve correlating intent recognition automation rates with intent recognition accuracy;

generating a continuous composite curve for the given domain by combining the input data set for the given domain with the input data sets for the curves of the sub-domains;

quantifying a correctness of the continuous composite curve by comparing the continuous composite curve to a related template curve corresponding to the domain;

responsive to the correctness being at least a given minimum level, determining that the continuous composite curve is an accurate representation of the input data set;

receiving an utterance from a user over a computer network;

determining an intent of the utterance and a corresponding confidence score for the determined intent;

determining whether the inferred intent is sufficiently accurate by comparing the determined confidence score to a confidence threshold for the continuous composite curve.

2. The computer-implemented method of claim 1 , further comprising:

based on the continuous composite curve, calculating, as the confidence threshold for the continuous composite curve, confidence threshold that provides a maximum intent recognition automation rate with at least a given minimum intent recognition accuracy.

3. The computer-implemented method of claim 1 , further comprising:

responsive to the determined confidence score being lower than the confidence threshold for the continuous composite curve, sending the utterance for analysis by human intent recognition.

4. The computer-implemented method of claim 1 , wherein the continuous composite curve corresponds to a given dialog state identifier (DSID) of a current point in a conversation, the computer-implemented method further comprising:

determining a DSID representing a current point in a conversation in which the utterance was uttered; and

determining to use the continuous composite curve for the utterance responsive at least in part to determining that the determined DSID matches the DSID corresponding to the continuous composite curve.

5. The computer-implemented method of claim 1 , further comprising generating the curves of the sub-domains, the generating comprising:

receiving utterances from users over a computer network;

inferring intent of the utterances using either natural language processing intent or human intent recognition as determined by a given automation rate;

determining true intents of the utterances by:

identifying differences between intents derived from natural language processing and intents selected by human intent analysts, and

using a human judge to resolve the identified differences.

6. The computer-implemented method of claim 1 , further comprising generating the curves of the sub-domains, the generating comprising:

receiving utterances from users over a computer network;

inferring intent of the utterances using either natural language processing intent or human intent recognition as determined by a given automation rate;

determining true intents of the utterances by:

identifying differences between intents derived from natural language processing and intents selected by human intent analysts, and

using intent analyst accuracy scores determined by certified quality assurance utterances to adjudicate differences.

7. The computer-implemented method of claim 1 , further comprising generating the curves of the sub-domains, the generating comprising:

receiving utterances from users over a computer network;

inferring intent of the utterances using either natural language processing intent or human intent recognition as determined by a given automation rate;

determining true intents of the utterances by:

identifying differences between intents derived from natural language processing and intents selected by human intent analysts, and

using transcribed utterances and natural language processing to determine accurate intents to adjudicate differences.

8. A non-transitory computer-readable storage medium storing instructions that when executed by a computer processor perform actions comprising:

obtaining, for a given sub-domain encompassed by a given domain, an input data set comprising a plurality of intent recognition automation rates and intent recognition accuracies corresponding to the automation rates;

obtaining, for each of a plurality of sub-domains disjoint from the given sub-domain and disjoint from each other and encompassed by the given domain, an input data set for a curve correlating intent recognition automation rates with intent recognition accuracy;

generating a continuous composite curve for the given domain by combining the input data set for the given domain with the input data sets for the curves of the sub-domains;

quantifying a correctness of the continuous composite curve by comparing the continuous composite curve to a related template curve corresponding to the domain;

responsive to the correctness being at least a given minimum level, determining that the continuous composite curve is an accurate representation of the input data set;

receiving an utterance from a user over a computer network;

determining an intent of the utterance and a corresponding confidence score for the determined intent;

determining whether the inferred intent is sufficiently accurate by comparing the determined confidence score to a confidence threshold for the continuous composite curve.

9. The non-transitory computer-readable storage medium of claim 8 , the actions further comprising:

based on the continuous composite curve, calculating, as the confidence threshold for the continuous composite curve, confidence threshold that provides a maximum intent recognition automation rate with at least a given minimum intent recognition accuracy.

10. The non-transitory computer-readable storage medium of claim 8 , the actions further comprising:

responsive to the determined confidence score being lower than the confidence threshold for the continuous composite curve, sending the utterance for analysis by human intent recognition.

11. The non-transitory computer-readable storage medium of claim 8 , wherein the continuous composite curve corresponds to a given dialog state identifier (DSID) of a current point in a conversation, the actions further comprising:

determining a DSID representing a current point in a conversation in which the utterance was uttered; and

determining to use the continuous composite curve for the utterance responsive at least in part to determining that the determined DSID matches the DSID corresponding to the continuous composite curve.

12. The non-transitory computer-readable storage medium of claim 8 , the actions further comprising generating the curves of the sub-domains, the generating comprising:

receiving utterances from users over a computer network;

inferring intent of the utterances using either natural language processing intent or human intent recognition as determined by a given automation rate;

determining true intents of the utterances by:

identifying differences between intents derived from natural language processing and intents selected by human intent analysts, and

using a human judge to resolve the identified differences.

13. The non-transitory computer-readable storage medium of claim 8 , the actions further comprising generating the curves of the sub-domains, the generating comprising:

receiving utterances from users over a computer network;

inferring intent of the utterances using either natural language processing intent or human intent recognition as determined by a given automation rate;

determining true intents of the utterances by:

identifying differences between intents derived from natural language processing and intents selected by human intent analysts, and

using intent analyst accuracy scores determined by certified quality assurance utterances to adjudicate differences.

14. The non-transitory computer-readable storage medium of claim 8 , the actions further comprising generating the curves of the sub-domains, the generating comprising:

receiving utterances from users over a computer network;

inferring intent of the utterances using either natural language processing intent or human intent recognition as determined by a given automation rate;

determining true intents of the utterances by:

identifying differences between intents derived from natural language processing and intents selected by human intent analysts, and

using transcribed utterances and natural language processing to determine accurate intents to adjudicate differences.

15. A computer system comprising:

a computer processor; and

a non-transitory computer-readable storage medium storing instructions that when executed by a computer processor perform actions comprising:

obtaining, for a given sub-domain encompassed by a given domain, an input data set comprising a plurality of intent recognition automation rates and intent recognition accuracies corresponding to the automation rates;

obtaining, for each of a plurality of sub-domains disjoint from the given sub-domain and disjoint from each other and encompassed by the given domain, an input data set for a curve correlating intent recognition automation rates with intent recognition accuracy;

generating a continuous composite curve for the given domain by combining the input data set for the given domain with the input data sets for the curves of the sub-domains;

quantifying a correctness of the continuous composite curve by comparing the continuous composite curve to a related template curve corresponding to the domain;

responsive to the correctness being at least a given minimum level, determining that the continuous composite curve is an accurate representation of the input data set;

receiving an utterance from a user over a computer network;

determining an intent of the utterance and a corresponding confidence score for the determined intent;

determining whether the inferred intent is sufficiently accurate by comparing the determined confidence score to a confidence threshold for the continuous composite curve.

16. The computer system of claim 15 , the actions further comprising:

based on the continuous composite curve, calculating, as the confidence threshold for the continuous composite curve, confidence threshold that provides a maximum intent recognition automation rate with at least a given minimum intent recognition accuracy.

17. The computer system of claim 15 , the actions further comprising:

responsive to the determined confidence score being lower than the confidence threshold for the continuous composite curve, sending the utterance for analysis by human intent recognition.

18. The computer system of claim 15 , wherein the continuous composite curve corresponds to a given dialog state identifier (DSID) of a current point in a conversation, the actions further comprising:

determining a DSID representing a current point in a conversation in which the utterance was uttered; and

determining to use the continuous composite curve for the utterance responsive at least in part to determining that the determined DSID matches the DSID corresponding to the continuous composite curve.

19. The computer system of claim 15 , the actions further comprising generating the curves of the sub-domains, the generating comprising:

receiving utterances from users over a computer network;

inferring intent of the utterances using either natural language processing intent or human intent recognition as determined by a given automation rate;

determining true intents of the utterances by:

identifying differences between intents derived from natural language processing and intents selected by human intent analysts, and

using a human judge to resolve the identified differences.

20. The computer system of claim 15 , the actions further comprising generating the curves of the sub-domains, the generating comprising:

receiving utterances from users over a computer network;

inferring intent of the utterances using either natural language processing intent or human intent recognition as determined by a given automation rate;

determining true intents of the utterances by:

identifying differences between intents derived from natural language processing and intents selected by human intent analysts, and

using intent analyst accuracy scores determined by certified quality assurance utterances to adjudicate differences.

Assignments (8)