Method and system for predicting speech recognition performance using accuracy scores
A system and method are presented for predicting speech recognition performance using accuracy scores in speech recognition systems within the speech analytics field. A keyword set is selected. Figure of Merit (FOM) is computed for the keyword set. Relevant features that describe the word individually and in relation to other words in the language are computed. A mapping from these features to FOM is learned. This mapping can be generalized via a suitable machine learning algorithm and be used to predict FOM for a new keyword. In at least embodiment, the predicted FOM may be used to adjust internals of speech recognition engine to achieve a consistent behavior for all inputs for various settings of confidence values.
1. A method for predicting speech recognition performance in a speech recognition system comprising a recognition engine, a database, a model learning module, and a performance prediction module, the method comprising:
determining, by the performance prediction module, at least one feature vector for an input into the speech recognition system;
creating a prediction model by:
selecting a set of keywords by:
extracting at least one feature from a plurality of feature vectors from at least one keyword from an initial keyword set, wherein the at least one feature is assigned a value;
determining a range of values of the at least one feature;
comparing the range of values of the at least one feature to an expected range of values of the at least one feature; and
adding at least one keyword to the initial keyword set in response to determining that the range of values differs from the expected range of values to generate the set of keywords;
learning the prediction model for predicting a figure of merit of the input by:
inputting the at least one feature vector and the set of keywords into the model learning module; and
adjusting parameters to minimize a cost function; and
applying, by the performance prediction module, the prediction model to predict a figure of merit for the speech recognition system based on the at least one feature vector;
reporting, by the performance prediction module, the predicted figure of merit for the speech recognition system performance; and
adjusting the recognition engine based on the predicted figure of merit.
2. The method of claim 1 , wherein the feature vector comprises at least one of: number of phonemes, number of syllables, number of stressed vowels.
3. The method of claim 1 , wherein the determining the range of values of the at least one feature comprises determining whether values of the at least one feature are: lower values, medium values or higher values.
4. The method of claim 1 , wherein the figure of merit is indicative of the accuracy of performance of the speech recognition system, wherein the figure of merit (fom) is predicted using a mathematical expression:
fom
=
∑
i
=
1
N
a
i
(
x
i
-
b
i
)
2
where N represents an upper limit on a number of features based on the determined feature vector used to learn the prediction, i represents the index of features, x i represents the i-th feature in the determined feature vector, and the equation parameters a and b are learned values.
5. The method of claim 1 , wherein the figure of merit prediction has a detection rate averaging 5 false alarms per keyword per hour of speech.
6. The method of claim 1 , wherein the method is performed in real-time as additional input is provided.
7. The method of claim 1 , wherein the determining the at least one feature vector comprises:
converting the input into a sequence of phonemes; and
performing morphological analysis of words in a language.
8. The method of claim 7 , wherein the converting is performed using statistics for phonemes and phoneme confusion matrix.
9. The method of claim 8 , further comprising computing the phoneme confusion matrix using a phoneme recognizer.
10. The method of claim 9 , further comprising automatically adjusting internal scores of the recognition engine based on the prediction reported by the performance prediction module.
11. A computer system comprising a processor and associated memory configured for executing software programs, the system being configured for predicting speech recognition performance, wherein the memory stores instructions that, when executed by the processor, cause the processor to:
use a performance prediction module to determine at least one feature vector for an input into a speech recognition system;
create a prediction model by:
selecting a set of keywords by:
extracting at least one feature from a plurality of feature vectors from at least one keyword from an initial keyword set, wherein the at least one feature is assigned a value;
determining a range of values of the at least one feature;
comparing the range of values of the at least one feature to an expected range of values of the at least one feature; and
adding at least one keyword to the initial keyword set in response to determining that the range of value differs from the expected range of values to generate the set of keywords;
learning the prediction model for predicting a figure of merit of the input by:
inputting at least one feature vector and the set of keywords into a model learning module; and
adjusting parameters to minimize a cost function, and
apply, by the performance prediction module, the prediction model to predict a figure of merit for the speech recognition system based on the at least one feature vector;
report, by the performance prediction module, the predicted figure of merit for the speech recognition system performance; and
adjust a recognition engine based on the predicted figure of merit.
12. The system of claim 11 , wherein the feature vector comprises at least one of: number of phonemes, number of syllables, number of stressed vowels.
13. The system of claim 11 , wherein the determining the range of values of the at least one feature comprises determining whether values of the at least one feature are: lower values, medium values or higher values.
14. The system of claim 11 , wherein the figure of merit is indicative of the accuracy of performance of the speech recognition system, wherein the figure of merit (fom) is predicted using a mathematical expression:
fom
=
∑
i
=
1
N
a
i
(
x
i
-
b
i
)
2
where N represents an upper limit on a number of features based on the determined feature vector used to learn the prediction, i represents the index of features, x i represents the i-th feature in the determined feature vector, and the equation parameters a and b are learned values.
15. The system of claim 11 , wherein the instructions are executed by the processor in real-time as additional input is provided.
16. The system of claim 11 , wherein the instructions to determine the at least one feature vector comprise instructions that cause the processor to:
convert the input into a sequence of phonemes; and
perform morphological analysis of words in a language.
17. The system of claim 16 , wherein the converting is performed using statistics for phonemes and phoneme confusion matrix.
18. The system of claim 17 , further comprising computing the phoneme confusion matrix using a phoneme recognizer.
19. The system of claim 11 , further comprising automatically adjusting internal scores of the recognition engine based on the prediction reported by the performance prediction module.