System and method for automatic filtering of test utterance mismatches in automatic speech recognition systems
View Patent ↗A system and method are presented for the automatic filtering of test utterance mismatches in automatic speech recognition (ASR) systems. Test data are evaluated for match between audio and text in a language-independent manner. Utterances having mismatch are identified and isolated for either removal or manual verification to prevent incorrect measurements of the ASR system performance. In an embodiment, contiguous stretches of low probabilities in every utterance are searched for and removed. Such segments may be intra-word or cross-word. In another embodiment, scores may be determined using log DNN probability for every word in each utterance. Words may be sorted in the order of the scores and those utterances containing the least word scores are removed.
1. A method for the automatic filtering of test utterance mismatches between a transcription and an audio signal in automatic speech recognition systems, the method comprising:
creating an alignment of the audio signal with the reference transcription;
determining Deep Neural Network probabilities for a senone sequence from the alignment;
identifying a contiguous stretch of senone probabilities that do not meet a threshold in each utterance;
locating a segment of near zero senone probabilities within the contiguous stretch; and
removing any of the utterances.
2. The method of claim 1 , wherein the test utterances comprise data containing longer utterances.
3. The segment of claim 1 , wherein the segment is of a predetermined length.
4. The segment of claim 3 , wherein the segment is intra-word.
5. The segment of claim 3 , wherein the segment is cross-word.
6. The method of claim 1 , wherein the automatic speech recognition system is language independent.
7. A method for the automatic filtering of test utterance mismatches between a transcription and an audio signal in automatic speech recognition systems, the method comprising:
creating an alignment of the audio signal with the reference transcription;
determining a score using log Deep Neural Network probabilities for every word in each utterance of the audio signal;
sorting the words in order of their associated scores; and
removing the utterances containing problematic words that do not meet a threshold.
8. Method of claim 7 , wherein the determining of a score is computed as an average of log of senone probabilities over a length of the word.
9. The method of claim 8 , wherein the determining is performed using the mathematical formula:
S
w
=
1
m
∑
t
=
1
m
log
p
senone
t
,
where m represents a number of senones in a word.
10. The method of claim 7 , wherein the determining of a score is computed as an average of scores of all phones in a word.
11. The method of claim 10 , wherein the score is determined as an average of log senone probabilities over the length of a phone.
12. The method of claim 10 , wherein the determining is performed using the mathematical formula:
S
w
=
1
n
∑
ph
=
1
n
1
m
ph
∑
t
=
1
m
ph
log
p
senone
t
,
where n represents a number of phones in a word and m ph represents a number of senones in a phone.
13. The method of claim 7 , wherein the determining of a score is computed as an average of scores of all phones in a word.
14. The method of claim 13 , wherein the determining is performed using the mathematical formula:
S
w
=
1
n
∑
ph
=
1
n
max
1
<
t
<
m
ph
(
log
p
senone
t
)
,
where n represents a number of phones in a word and m ph represents a number of senones in a phone.
15. The method of claim 13 , where a phone score is determined as a maximum of log senone probabilities in a phone.
16. The method of claim 7 , wherein the automatic speech recognition system is language independent.
17. A system for the automatic filtering of test utterance mismatches between a transcription and an audio signal in automatic speech recognition systems, the system comprising:
a processor; and
a memory in communication with the processor, the memory storing instructions that, when executed by the processor, causes the processor to:
create an alignment of the audio signal with the reference transcription,
determine Deep Neural Network probabilities for a senone sequence from the alignment,
identify a contiguous stretch of senone probabilities that do not meet a threshold in each utterance,
locate a segment of near zero senone probabilities within the contiguous stretch, and
remove any of the utterances.
18. A system for the automatic filtering of test utterance mismatches between a transcription and an audio signal in automatic speech recognition systems, the system comprising:
a processor; and
a memory in communication with the processor, the memory storing instructions that, when executed by the processor, causes the processor to:
create an alignment of the audio signal with the reference transcription;
determine a score using log Deep Neural Network probabilities for every word in each utterance of the audio signal;
sort the words in order of their associated scores; and
remove the utterances containing problematic words that do not meet a threshold.