System and method for unsupervised and active learning for automatic speech recognition
View Patent ↗A system and method is provided for combining active and unsupervised learning for automatic speech recognition. This process enables a reduction in the amount of human supervision required for training acoustic and language models and an increase in the performance given the transcribed and un-transcribed data.
1. A method of training an automatic speech recognition module, the method comprising:
training, via a processor of a computing device, acoustic and language models using a set of transcribed data, speech recognition scores, and word confidence scores for a retrieved set of un-transcribed data, wherein the set of transcribed data, speech recognition scores, and word confidence scores is generated by steps comprising:
recognizing utterances in a set of candidates for transcription using acoustic and language models trained using an initial set of transcribed data;
computing confidence scores of the utterances;
selecting a subset of utterances that have the smallest confidence scores from the set of candidates and transcribing them into a transcribed set;
adding the transcribed set to the initial set of transcribed data to produce an updated set of transcribed data, speech recognition scores, and word confidence scores;
selecting a pre-determined amount of un-transcribed data, the un-transcribed data being associated with utterances not selected in the selecting of the subset of utterances; and
applying the pre-determined amount of un-transcribed data to train the acoustic and language models; and
iteratively performing the training if word accuracy has not converged.
2. The method of claim 1 , the method further comprising:
prior to the training, training the acoustic and language models using another set of transcribed data.
3. The method of claim 2 , further comprising:
recognizing utterances in the set of candidates for transcription using the acoustic and language models.
4. The method of claim 3 , further comprising:
computing by a processor confidence scores of the utterances.
5. The method of claim 4 , further comprising:
selecting k utterances that have smallest confidence scores from the set of candidates and transcribing the k utterances into a first additional transcribed set.
6. The method of claim 5 , wherein k is more than one.
7. The method of claim 5 , wherein selecting k utterances further comprises leaving out utterances with confidence scores indicating that the utterances were correctly recognized.
8. The method of claim 5 , further comprising:
adding the first additional transcribed set to the another set of transcribed data to produce the set of transcribed data.
9. The method of claim 8 , further comprising:
removing the first additional transcribed set from the set of candidates.
10. The method of claim 9 , wherein the retrieved set of un-transcribed data is retrieved from the set of candidates.
11. The method of claim 10 , further comprising selecting a sample of un-transcribed data.
12. The method of claim 1 , wherein word posterior probability estimates are used for word confidence scores associated with the utterances.
13. The method of claim 1 , wherein a word is considered to be correctly recognized if the word has a confidence score higher than a threshold value.
14. A tangible computer-readable medium that stores a program which, upon execution on a processor, causes the processor to train an automatic speech recognition module, the program comprising instructions for:
training acoustic and language models using a set of transcribed data, speech recognition scores and word confidence scores for a retrieved set of un-transcribed data, wherein the set of transcribed data, speech recognition scores, and word confidence scores is generated by steps comprising:
recognizing utterances in a set of candidates for transcription using acoustic and language models trained using an initial set of transcribed data;
computing confidence scores of the utterances;
selecting a subset of utterances that have the smallest confidence scores from the set of candidates and transcribing them into a transcribed set;
adding the transcribed set to the initial set of transcribed data to produce an updated set of transcribed data, speech recognition scores, and word confidence scores;
selecting a pre-determined amount of un-transcribed data, the un-transcribed data being associated with utterances not selected in the selecting of the subset of utterances; and
applying the pre-determined amount of un-transcribed data to train the acoustic and language models; and
iteratively performing the training if word accuracy has not converged.
15. The tangible computer-readable medium of claim 14 , the program further comprising instructions for, prior to the training, training the acoustic and language models using another set of transcribed data, recognizing utterances in a set of candidates for transcription using the acoustic and language models and computing by a processor confidence scores of the utterances.
16. The tangible computer-readable medium of claim 15 , the program further comprising instructions for selecting k utterances that have the smallest confidence scores from the set of candidates and transcribing them into a first additional transcribed set, adding the first additional transcribed set to the another set of transcribed data to produce the set of transcribed data and removing the first additional transcribed set from the set of candidates and wherein the retrieved set of un-transcribed data is retrieved from the set of candidates.
17. A spoken dialog system, the system comprising:
a processor;
an automatic-speech recognition module controlling the processor to perform automatic speech recognition, the automatic speech recognition module trained using a method of training an automatic speech recognition module and stored in a memory storage device, the method comprising:
training acoustic and language models using a set of transcribed data, speech recognition scores and word confidence scores for a retrieved set of un-transcribed data, wherein the set of transcribed data, speech recognition scores, and word confidence scores is generated by steps comprising:
recognizing utterances in a set of candidates for transcription using acoustic and language models trained using an initial set of transcribed data;
computing confidence scores of the utterances;
selecting a subset of utterances that have the smallest confidence scores from the set of candidates and transcribing them into a transcribed set;
adding the transcribed set to the initial set of transcribed data to produce an updated set of transcribed data, speech recognition scores, and word confidence scores;
selecting a pre-determined amount of un-transcribed data, the un-transcribed data being associated with utterances not selected in the selecting of the subset of utterances; and
applying the pre-determined amount of un-transcribed data to train the acoustic and language models; and
iteratively performing the training if word accuracy has not converged.
18. The spoken dialog system of claim 17 , wherein the method further comprises, prior to the training, training the acoustic and language models using another set of transcribed data, recognizing utterances in a set of candidates for transcription using the acoustic and language models and computing by a processor confidence scores of the utterances.
19. The spoken dialog system of claim 18 , wherein the method further comprises selecting k utterances that have the smallest confidence scores from the set of candidates and transcribing them into a first additional transcribed set, adding the first additional transcribed set to the another set of transcribed data to produce the set of transcribed data and removing the first additional transcribed set from the set of candidates and wherein the retrieved set of un-transcribed data is retrieved from the set of candidates.