Active labeling for spoken language understanding
View Patent ↗A spoken language understanding method and system are provided. The method includes classifying a set of labeled candidate utterances based on a previously trained classifier, generating classification types for each candidate utterance, receiving confidence scores for the classification types from the trained classifier, sorting the classified utterances based on an analysis of the confidence score of each candidate utterance compared to a respective label of the candidate utterance, and rechecking candidate utterances according to the analysis. The system includes modules configured to control a processor in the system to perform the steps of the method.
1. A spoken language understanding system comprising:
a processor;
a first module configured to control the processor to classify a set of labeled candidate utterances based on a previously trained classifier and generate classification types for each candidate utterance, each classification type having a respective confidence score;
a second module configured to control the processor to sort the candidate utterances based on an analysis of the confidence score of each candidate utterance compared to a respective label of the candidate utterance, wherein the analysis is based at least in part on a Kullback-Liebler divergence; and
a third module configured to control the processor to recheck candidate utterances according to when the Kullback-Liebler divergence is greater than a threshold.
2. The system of claim 1 , wherein the confidence scores are generated for a plurality of labels associated with the classification types.
3. The system of claim 1 , wherein the confidence scores are received from the previously trained classifier.
4. The system of claim 1 , wherein the analysis is based on a comparison of the generated classification type of the candidate utterance and the label of the candidate utterance, and the module rechecks the candidate utterances when the generated classification type does not match the label of the candidate utterance.
5. The system of claim 1 , wherein the Kullback-Leibler divergence is based on the computation:
KL
(
P
Q
)
=
∑
i
∈
L
p
i
×
log
(
p
i
q
i
)
+
(
1
-
p
i
)
×
log
(
1
-
p
i
1
-
q
i
)
where L is the set of all classification types, q i is the probability of the i th classification type obtained from the trained classifier, and p=1 if that classification type is previously labeled and p=0 if otherwise.
6. A tangible computer-readable medium storing instructions for controlling a computer device, as part of a spoken language understanding system, to perform the steps of:
classifying a set of labeled candidate utterances based on a previously trained classifier;
generating classification types for each candidate utterance, each classification type having a respective confidence score;
sorting the candidate utterances based on an analysis of the confidence score of each candidate utterance compared to a respective label of the candidate utterance, wherein the analysis is based at least in part on a Kullback-Liebler divergence; and
rechecking the candidate utterances that do not match the label of the respective candidate utterance when the Kullback-Liebler divergence is greater than a threshold.
7. The tangible computer-readable medium of claim 6 , further comprising generating confidence scores for a plurality of labels associated with the classification types.
8. The tangible computer-readable medium of claim 6 , further comprising receiving confidence scores from the previously trained classifier.
9. The tangible computer-readable medium of claim 6 , further comprising identifying candidate utterances having a top choice of classification types that does not match the label of the respective candidate utterances.
10. A computer-implemented method for automatic speech recognition that extracts words from user speech, the system comprising:
classifying, via a processor, a set of labeled candidate utterances based on a previously trained classifier;
generating, via a processor, classification types for each candidate utterance;
receiving confidence scores for the classification types from the trained classifier;
sorting the classified utterances based on an analysis of the confidence score of each candidate utterance compared to a respective label of the candidate utterance, wherein the analysis is based at least in part on a Kullback-Liebler divergence; and
rechecking candidate utterances when the Kullback-Liebler divergence is greater than a threshold.
11. The computer-implemented method of claim 10 , wherein the analysis is based on a comparison of the generated classification type of the candidate utterance and the label of the candidate utterance, and the module rechecks the candidate utterances when the generated classification type does not match the label of the candidate utterance.
12. The computer-implemented method of claim 10 , wherein the Kullback-Leibler divergence is based on the computation:
KL
(
P
Q
)
=
∑
i
∈
L
p
i
×
log
(
p
i
q
i
)
+
(
1
-
p
i
)
×
log
(
1
-
p
i
1
-
q
i
)
where L is the set of all classification types, q i is the probability of the i th classification type obtained from the trained classifier, and p=1 if that classification type is previously labeled and p=0 if otherwise.
13. The computer-implemented method of claim 10 , wherein the classification types include call classification types.
14. The computer-implemented method of claim 10 , wherein the trained classifier comprises an iterative classifier.