Active learning for spoken language understanding
View Patent ↗Disclosed is a system and method of training a spoken language understanding module. Such a module may be utilized in a spoken dialog system. The method of training a spoken language understanding module comprises training acoustic and language models using a small set of transcribed data S T , recognizing utterances in a set S u that are candidates for transcription using the acoustic and language models, computing confidence scores of the utterances, selecting k utterances that have the smallest confidence scores from S u and transcribing them into a new set S i , redefining S t as the union of S t and S i , redefining S u as S u minus S i , and returning to the step of training acoustic and language models if word accuracy has not converged.
1. A method generating a classifier from training data S t and a larger amount of unlabeled data in a pool S u , the method comprising:
(1) training a classifier using current training data S t , the training data S t generated by sampling a plurality of utterances;
(2) classifying utterances in a pool S u using the trained classifier;
(3) computing a call type confidence score for each utterance;
(4) sorting candidate utterances with respect to the confidence score of the maximum scoring call type;
(5) selecting the lowest scored k utterances from S u using the confidence scores and labeling them to define a labeled set S i ;
(6) redefining S t =S t ∪S i ; and
(7) redefining S u =S u −S i .
2. The method of claim 1 , wherein steps 1 through 7 are practiced until labelers and utterances are no longer available.
3. The method of claim 1 , wherein k is more than one.
4. The method of claim 1 , wherein selecting k utterances from S u further comprises leaving out utterances with confidence scores indicating that the utterances were correctly recognized.
5. The method of claim 1 , wherein selecting k utterances from S u further comprises selecting the lowest scoring k utterances from S u .
6. The method of claim 1 , wherein selecting k utterances from S u further comprises selecting utterances according to a confidence score distribution that is closest to a prior distribution.
7. A system having spoken language understanding module generated according to a method comprising:
(1) training a plurality of classifiers independently using a training data set S t , the training data S t generated by sampling a plurality of utterances;
(2) classifying utterances in a set S u using the plurality of classifiers and computing a call type confidence score for all utterances;
(3) sorting candidate utterances with respect to a score of the maximum scoring call type according to one of the classifiers if the classifiers disagree;
(4) selecting and labeling the lowest scored k utterances from S u to define a labeled set S i and redefining S t and S u as follows:
(5) S t =S t ∪S i ; and
(6) S u =S u −S t , wherein the labeled utterances are used to generate the spoken language understanding module.
8. The system of claim 7 , wherein selecting k utterances from S u further comprises selecting the lowest scoring k utterances from S u .
9. The system of claim 7 , wherein the method further comprises performing steps 1-6 while labelers and utterances are available.
10. A system having spoken language understanding module trained using a method comprising:
(1) training acoustic and language models using a small set of transcribed data S t , the training data S t generated by sampling a plurality of utterances;
(2) recognizing utterances in a set S u that are candidates for transcription using the acoustic and language models;
(3) computing confidence scores of the utterances;
(4) selecting the lowest scored k utterances from S u using the confidence scores and transcribing them into a new set S i ;
(5) redefining S t as the union of S t and S i ;
(6) redefining S u as S u minus S i ; and
(7) returning to step (1) of word accuracy has not converged.
11. The system of claim 10 , wherein selecting k utterances from S u further comprises selecting utterances according to a confidence score distribution that is closest to a prior distribution.
12. The system of claim 10 , wherein selecting k utterances from S u further comprises selecting the lowest scoring k utterances from S u .
13. A method of generating a spoken language understanding module, the method comprising, from small amount of training data S t and a larger amount of unlabeled data S u :
(1) training a plurality a classifiers independently using a training data set S i , the training data S t generated by sampling a plurality of utterances;
(2) classifying utterances in a set S u using the plurality of classifiers and computing a call type confidence score for all utterances;
(3) sorting candidate utterances with respect to a score of the maximum scoring call type according to one of the classifiers if the classifiers disagree;
(4) selecting and labeling the lowest scored k utterances from S u to define a labeled set S i and redefining S t and S u as follows:
(5) S t =S t ∪S i ; and
(6) S u =S u −S i , wherein the labeled utterances are used to generate the spoken language understanding module.
14. The method of claim 13 , wherein the steps occur only while labelers and utterances are available.
15. The method of claim 13 , wherein selecting k utterances from S u further comprises selecting utterances according to a confidence score distribution that is closet to a prior distribution.
16. The method of claim 13 , wherein selecting k utterances from S u further comprises selecting the lowest scoring k utterances from S u .