Automatic methods to predict error rates and detect performance degradation

1. A computer-implemented method comprising:

generating, by a computing system and utilizing a set of speech-recognition models, text corresponding to input speech spoken by a user during a first dictation session;

determining, by the computing system, based on the text corresponding to the input speech spoken by the user during the first dictation session, and without comparing the text corresponding to the input speech spoken by the user during the first dictation session to preexisting text corresponding to the input speech spoken by the user during the first dictation session, a metric correlated to a verbatim-recognition error rate of the first dictation session;

generating, by the computing system and utilizing an updated set of speech-recognition models, text corresponding to input speech spoken by the user during a second dictation session;

determining, by the computing system, based on the text corresponding to the input speech spoken by the user during the second dictation session, and without comparing the text corresponding to the input speech spoken by the user during the second dictation session to preexisting text corresponding to the input speech spoken by the user during the second dictation session, a metric correlated to a verbatim-recognition error rate of the second dictation session; and

comparing, by the computing system, the metric correlated to the verbatim-recognition error rate of the first dictation session with the metric correlated to the verbatim-recognition error rate of the second dictation session to determine if recognition performance degraded for the user during the second dictation session due to utilization of the updated set of speech-recognition models.

2. The computer-implemented method of claim 1 , wherein determining the metric correlated to the verbatim-recognition error rate of the first dictation session comprises determining a word-level confidence metric for the first dictation session, and wherein determining the metric correlated to the verbatim-recognition error rate of the second dictation session comprises determining a word-level confidence metric for the second dictation session.

3. The computer-implemented method of claim 2 , wherein determining the word-level confidence metric for the first dictation session comprises determining, using a logistic regression model, an average word confidence for the first dictation session, and wherein determining the word-level confidence metric for the second dictation session comprises determining, using the logistic regression model, an average word confidence for the second dictation session.

4. The computer-implemented method of claim 3 , wherein determining the average word confidence for the first dictation session comprises, determining, for each of a plurality of features, a logistic regression for the first dictation session, and wherein determining the average word confidence for the second dictation session comprises, determining, for each of the plurality of features, a logistic regression for the second dictation session.

5. The computer-implemented method of claim 4 , wherein:

determining the average word confidence for the first dictation session comprises:

determining a plurality of acoustic-model scores for the first dictation session;

determining a plurality of language-model scores for the first dictation session;

determining a logistic regression of the plurality of acoustic-model scores for the first dictation session; and

determining a logistic regression of the plurality of language-model scores for the first dictation session; and

determining the average word confidence for the second dictation session comprises:

determining a plurality of acoustic-model scores for the second dictation session;

determining a plurality of language-model scores for the second dictation session;

determining a logistic regression of the plurality of acoustic-model scores for the second dictation session; and

determining a logistic regression of the plurality of language-model scores for the second dictation session.

6. The computer-implemented method of claim 3 , comprising calibrating, after determining the average word confidence for the first dictation session, before determining the average word confidence for the second dictation session, and based on a dataset associated with the user and a posterior probability associated with the user, the logistic regression model.

7. A system comprising:

at least one processor; and

a memory storing instructions that when executed by the at least one processor cause the system to:

determine metrics correlated to verbatim-recognition error rates of distinct dictation sessions without comparing text generated by the system, using differing sets of speech-recognition models, for input speech spoken by a user during the distinct dictation sessions to preexisting text corresponding to the input speech; and

compare one or more metrics of the metrics with one or more other metrics of the metrics to determine whether recognition performance degraded for the user between one or more of the distinct dictation sessions due to utilization of one or more of the differing sets of speech-recognition models.

8. The system of claim 7 , wherein the instructions, when executed by the at least one processor, cause the system to determine, for each of the distinct dictation sessions, a word-level confidence metric.

9. The system of claim 7 , wherein the instructions, when executed by the at least one processor, cause the system to determine, for each of the distinct dictation sessions and based on a logistic regression model, an average word confidence.

10. The system of claim 9 , wherein the instructions, when executed by the at least one processor, cause the system to calibrate, between the distinct dictation sessions and based on a dataset associated with the user and a posterior probability associated with the user, the logistic regression model.

11. The system of claim 7 , wherein the instructions, when executed by the at least one processor, cause the system to determine, for each dictation session of the distinct dictation sessions, a logistic regression of each of a plurality of features of the dictation session.

12. The system of claim 7 , wherein the instructions, when executed by the at least one processor, cause the system to determine, for each of the distinct dictation sessions, a plurality of acoustic-model scores, a plurality of language-model scores, a logistic regression of the plurality of acoustic-model scores, and a logistic regression of the plurality of language-model scores.

13. The system of claim 7 , wherein the instructions, when executed by the at least one processor, cause the system to:

determine a difference between the metrics; and

compare the difference to a predetermined threshold.

14. One or more non-transitory computer-readable media having instructions stored thereon that when executed by one or more computers cause the one or more computers to compare, against one another, metrics, correlated to verbatim-recognition error rates of distinct dictation sessions, determined by the one or more computers without comparing text generated by the one or more computers, using differing sets of speech-recognition models, for input speech spoken by a user during the distinct dictation sessions to preexisting text corresponding to the input speech, to determine whether recognition performance degraded for the user between one or more of the distinct dictation sessions due to utilization of one or more of the differing sets of speech-recognition models.

15. The one or more non-transitory computer-readable media of claim 14 , wherein the instructions, when executed by the one or more computers, cause the one or more computers to determine, for each of the distinct dictation sessions, a word-level confidence metric.

16. The one or more non-transitory computer-readable media of claim 14 , wherein the instructions, when executed by the one or more computers, cause the one or more computers to determine, for each of the distinct dictation sessions and based on a logistic regression model, an average word confidence.

17. The one or more non-transitory computer-readable media of claim 16 , wherein the instructions, when executed by the one or more computers, cause the one or more computers to calibrate, between the distinct dictation sessions and based on a dataset associated with the user and a posterior probability associated with the user, the logistic regression model.

18. The one or more non-transitory computer-readable media of claim 14 , wherein the instructions, when executed by the one or more computers, cause the one or more computers to determine, for each dictation session of the distinct dictation sessions, a logistic regression of each of a plurality of features of the dictation session.

19. The one or more non-transitory computer-readable media of claim 14 , wherein the instructions, when executed by the one or more computers, cause the one or more computers to determine, for each of the distinct dictation sessions, a plurality of acoustic-model scores, a plurality of language-model scores, a logistic regression of the plurality of acoustic-model scores, and a logistic regression of the plurality of language-model scores.

20. The one or more non-transitory computer-readable media of claim 14 , wherein the instructions, when executed by the one or more computers, cause the one or more computers to:

determine a difference between the metrics; and

compare the difference to a predetermined threshold.