IP Library Granted Patent US 10,431,214
Granted Patent B2
US 10,431,214 · App. 14/951,389 · Granted Oct 1, 2019

System and method of determining a domain and/or an action related to a natural language input

Inventors: Yikun Guo (Bellevue, WA); Safiyyah Saleem (Bellevue, WA); Jiayuan Huang (Bellevue, WA)
Assignee: Voicebox Technologies Corporation
G10L15/22G06F17/27G10L15/1815
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,431,214
App. No.
14/951,389
Granted
Oct 1, 2019
Kind
B2
Abstract

The disclosure relates to methods, systems and other embodiments directed to determining an information domain match for a natural language (NL) input (e.g., a spoken utterance), and confirming whether the NL input is correctly matched to the information domain. For example, after receiving an NL input, a first information domain to which the NL input belongs and a feature value set may be determined based on a semantic pattern matching technique. Further, a second information domain to which the NL input belongs, and a corresponding confidence score related to the second information domain may be determined. The second information domain may be determined based on a first statistical classification technique. Based on the determined feature value set and the confidence score related to the second information domain, it may be confirmed whether the NL input correctly belongs to the first information domain, e.g., based on a second statistical classification technique.

Claims (41)

1. A computer-implemented method to confirm whether a domain determined for a natural language input using a semantic pattern matching technique is correct based on outputs of the semantic pattern matching technique and a domain determined for the natural language input and outputs using statistics-based natural language processing, the method being implemented on a computer system having one or more physical processors programmed with computer program instructions which, when executed, perform the method, the method comprising:

receiving, at the computer system, a natural language (NL) input;

determining, by the computer system, based on the semantic pattern matching technique, a first information domain for the NL input, and a feature value set comprising information indicating a first confidence score from the semantic pattern matching technique, the first confidence score indicating a level of confidence that the first information domain for the NL input determined by the semantic pattern matching technique is correct;

determining, by the computer system, based on the statistics-based natural language processing, a second information domain for the NL input, and a second confidence score indicating a level of confidence that the second information domain for the NL input determined by the statistics-based natural language processing is correct;

providing, by the computer system, the first information domain determined by the semantic pattern matching technique, the feature value set, the second information domain determined by the statistics-based natural language processing, and the second confidence score as input to a classifier engine;

determining, by the computer system, using the classifier engine, a third confidence score different from both the first confidence score and the second confidence score, that indicates a level of confidence that the first information domain determined by the semantic pattern matching technique has been confirmed as correctly identified based on the first information domain determined by the semantic pattern matching technique, the feature value set, the second information domain determined by the statistics-based natural language processing, and the second confidence score;

comparing, by the computer system, the third confidence score with a predefined threshold confidence score that indicates the first information domain determined by the semantic pattern matching technique is confirmed as correctly identified; and

determining, by the computer system, whether the first information domain determined by the semantic pattern matching technique is confirmed based on the comparison.

2. The method of claim 1 , wherein the NL input is represented based on one or more tokens, and wherein said determining based on the semantic pattern matching technique comprises matching, by the computer system, based at least on a semantic grammar, the one or more tokens to one or more computer-executable tasks and one or more entities, wherein the one or more tasks and the one or more entities are predefined for the semantic pattern matching technique.

3. The method of claim 2 , wherein the feature value set comprises one or more of the following: a. total number of the one or more tokens matched, a fraction number of the one or more tokens matched, a matching score assigned to at least one of the one or more tokens, and a matching score assigned to the NL input.

4. The method of claim 1 , wherein the NL input is represented based on one or more tokens, the method further comprising generating, by the computer system, an input feature vector for the NL input, wherein each element of the input feature vector represents at least one of the one or more tokens, wherein said determining the second information domain determined by the statistics-based natural language processing and the second confidence score comprises processing, by the computer system, the input feature vector based on a first statistical classification technique.

5. The method of claim 4 , wherein said processing the input feature vector comprises processing, by the computer system, the input feature vector based on the first statistical classification technique to determine a plurality of second information domains for the NL input and a corresponding plurality of confidence scores, wherein each corresponding confidence score of the corresponding plurality of confidence scores is indicative of the likelihood of the corresponding one of the plurality of second information domains being correct.

6. The method of claim 4 , further comprising processing, by the computer system, the input feature vector based on the first statistical classification technique to determine a binary check value indicative of whether the second information domain determined by the statistics-based natural language processing is related to one of a plurality of information domains associated with the semantic pattern matching technique, wherein said determining whether the NL input for the first information domain determined by the semantic pattern matching technique is confirmed, is further based on the binary check value.

7. The method of claim 4 , wherein the first statistical classification technique is based on one or more of the following:

a maximum-entropy classifier, a logistic regression classifier, a Naive B ayes classifier, and a support vector machine.

8. The method of claim 1 , wherein said determining whether the first information domain determined by the semantic pattern matching technique is confirmed, comprises processing, by the computer system, the feature value set and the second confidence score based on a second statistical classification technique.

9. The method of claim 8 , wherein said processing the feature value set and the second confidence score based on the second statistical classification technique comprises:

obtaining, by the computer system, coefficient weights associated with the second statistical classification technique for the feature value set and the second confidence score; and

processing, by the computer system, the feature value set and the second confidence score with the coefficient weights based on the second statistical classification technique to determine the third confirmation score.

10. The method of claim 1 , wherein the NL input comprises a spoken NL utterance.

11. A system to confirm whether a domain determined for a natural language input using a semantic pattern matching technique is correct based on outputs of the semantic pattern matching technique and a domain determined for the natural language input and outputs using statistics-based natural language processing, the system comprising:

one or more physical processors programmed with computer program instructions which, when executed, cause the one or more physical processors to:

receive a natural language (NL) input;

determine, based on the semantic pattern matching technique, a first information domain for the NL input, and a feature value set comprising information indicating first confidence score from the semantic pattern matching technique, the first confidence score indicating a level of confidence that the first information domain for the NL input determined by the semantic pattern matching technique is correct;

determine, based on the statistics-based natural language processing, a second information domain for the NL input, and a second confidence score indicating a level of confidence that the second information domain for the NL input determined by the statistics-based natural language processing is correct;

provide the first information domain determined by the semantic pattern matching technique, the feature value set, the second information domain determined by the statistics-based natural language processing, and the second confidence score as input to a classifier engine;

determine, using the classifier engine, a third confidence score that indicates a level of confidence that the first information domain determined by the semantic pattern matching technique has been confirmed as correctly identified based on the first information domain determined by the semantic pattern matching technique, the feature value set, the second information domain determined by the statistics-based natural language processing, and the second confidence score;

compare the third confidence score with a predefined threshold confidence score that indicates the first information domain determined by the semantic pattern matching technique is confirmed as correctly identified; and

determine whether the first information domain determined by the semantic pattern matching technique is confirmed based on the comparison.

12. The system of claim 11 , wherein the NL input is represented using one or more tokens, and wherein, to determine based on the semantic pattern matching technique, the one or more physical processors are further caused to match, based at least on a semantic, grammar, the one or more tokens to one or more computer-executable tasks and one or more entities, wherein the one or more tasks and the one or more entities are predefined for the semantic pattern matching technique.

13. The system of claim 12 , wherein the feature value set comprises one or more of the following: a total number of the one or more tokens matched, a fraction number of the one or more tokens matched, a matching score assigned to at least one of the one or more tokens, and a matching score assigned to the NL input.

14. The system of claim 11 , wherein the NL input is represented based on one or more tokens, and wherein the one or more physical processors are further caused to:

generate an input feature vector for the NL input, wherein each element of the input feature vector represents at least one of the one or more tokens, and to determine the second information domain determined by the statistics-based natural language processing and the second confidence score, process the input feature vector based on a first statistical classification technique.

15. The system of claim 14 , wherein, to process the input feature vector, the one or more physical processors are further caused to process the input feature vector based on the first statistical classification technique to determine a plurality of second information domains for the NL input and a corresponding plurality of confidence scores, wherein each corresponding confidence score of the corresponding plurality of confidence scores is indicative of the likelihood of the corresponding one of the plurality of second information domains being correct.

16. The system of claim 14 , wherein the one or more physical processors are further caused to process the input feature vector based on the first statistical classification technique to determine a binary check value indicative of whether the second information domain determined by the statistics-based natural language processing is related to one of a plurality of information domains associated with the semantic pattern matching technique, wherein, a determination of whether the NL, input for the first information domain determined by the semantic pattern matching technique is confirmed, is further based on the binary check value.

17. The system of claim 14 , wherein the first statistical classification technique is based on one or more of the following: a maximum-entropy classifier, a logistic regression classifier, a Naive Bayes classifier, and a support vector machine.

18. The system of claim 11 , wherein, to determine whether the first information domain determined by the semantic pattern matching technique is confirmed, comprises processing, by the computer system, the feature value set and the second confidence score based on a second statistical classification technique.

19. The system of claim 18 , wherein, to process the feature value set and the second confidence score based on the second statistical classification technique, the one or more physical processors are further caused to:

obtain coefficient weights associated with the second statistical classification technique for the feature value set and the second confidence score; and

process the feature value set and the second confidence score with the coefficient weights based on the second statistical classification technique to determine the third confirmation score.

20. The system of claim 11 , wherein the NL input comprises a spoken NL, utterance.

Assignments (8)
RELEASE (REEL 052935 / FRAME 0584) Recorded Jan 2, 2025
From: WELLS FARGO BANK, NATIONAL ASSOCIATION
To: CERENCE OPERATING COMPANY
Reel/Frame 069797/0818 →
SECURITY AGREEMENT Recorded Jun 15, 2020
From: CERENCE OPERATING COMPANY
To: WELLS FARGO BANK, N.A.
Reel/Frame 052935/0584 →
RELEASE OF SECURITY INTEREST Recorded Jun 12, 2020
From: BARCLAYS BANK PLC
To: CERENCE OPERATING COMPANY
Reel/Frame 052927/0335 →
SECURITY AGREEMENT Recorded Nov 7, 2019
From: CERENCE OPERATING COMPANY
To: BARCLAYS BANK PLC
Reel/Frame 050953/0133 →
INTELLECTUAL PROPERTY AGREEMENT Recorded Oct 24, 2019
From: NUANCE COMMUNICATIONS, INC.
To: CERENCE INC.
Reel/Frame 050818/0001 →
RELEASE OF SECURITY INTEREST Recorded Apr 5, 2018
From: ORIX GROWTH CAPITAL, LLC
To: VOICEBOX TECHNOLOGIES CORPORATION
Reel/Frame 045581/0630 →
SECURITY INTEREST Recorded Dec 22, 2017
From: VOICEBOX TECHNOLOGIES CORPORATION
To: ORIX GROWTH CAPITAL, LLC
Reel/Frame 044949/0948 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 20, 2016
From: GUO, YIKUN; SALEEM, SAFIYYAH; HUANG, JIAYUAN
To: VOICEBOX TECHNOLOGIES CORPORATION
Reel/Frame 038958/0378 →
Continuity (2)
Provisional Application 62085029 · Nov 26, 2014
Related Publication 20160148612A1 · May 26, 2016
Cited By (3)
US 12,424,110 US 12,511,480 US 12,665,093