KPI-threshold selection for audio-transcription models
In general, this disclosure describes techniques for generating and evaluating automatic transcripts of audio recordings containing human speech. In some examples, a computing system is configured to: generate transcripts of a plurality of audio recordings; determine an error rate for each transcript by comparing the transcript to a reference transcript of the audio recording; receive, for each transcript, a subjective ranking selected from a plurality of subjective rank categories; determine, based on the error rates and subjective rankings, objective rank categories defined by error-rate ranges; and assign an objective ranking to a new machine-generated transcript of a new audio recording, based on the objective rank categories and an error rate of the new machine-generated transcript.
1. A method comprising:
generating, by a computing system and for each of a plurality of audio recordings, a machine-generated transcript of the audio recording;
identifying, by the computing system and for each of the audio recordings, an error rate for each of the machine-generated transcripts, wherein each error rate is generated by comparing the machine-generated transcript to a reference transcript of the audio recording;
receiving, by the computing system and for each of the machine-generated transcripts, a subjective ranking selected from a plurality of subjective rank categories;
determining, by the computing system and based on the error rates and subjective rankings for the plurality of audio recordings, objective rank categories defined by error-rate ranges; and
assigning, by the computing system and to a new machine-generated transcript of a new audio recording not included in the plurality of audio recordings, an objective ranking based on the objective rank categories and an error rate of the new machine-generated transcript.
2. The method of claim 1 , wherein the subjective rank categories comprise three categories associated with progressively good transcript quality.
3. The method of claim 1 , wherein each of the objective rank categories corresponds to one of the subjective rank categories.
4. The method of claim 1 , wherein determining the objective rank categories comprises selecting error-rate boundaries between any two consecutive, non-overlapping subjective rank categories based on a difference between each error rate and the error-rate boundary.
5. The method of claim 1 , wherein determining the objective rank categories comprises selecting error-rate boundaries between any two consecutive, overlapping subjective rank categories based on:
the difference between each error rate and the error-rate boundary; and
a relative proportion of error rates sharing a common subjective rank category and falling on a common side of the boundary.
6. The method of claim 1 , wherein determining the objective rank categories comprises applying a support vector machine (SVM) algorithm to the error rates and to the subjective rankings of the machine-generated transcripts in order to select error-rate boundaries between the error-rate ranges.
7. The method of claim 1 , further comprising determining, based on the objective rankings of a plurality of new machine-generated transcripts not included in the machine-generated transcripts, a relative accuracy of an audio-transcription application configured to generate the plurality of new machine-generated transcripts.
8. The method of claim 1 , wherein identifying the error rate for each machine-generated transcript comprises identifying the error rate based on a percent of incorrect and missed words in the machine-generated transcript relative to the reference transcript.
9. The method of claim 8 , wherein identifying the error rate for each machine-generated transcript further comprises identifying the error rate based on a weighting indicating the relative importance of each incorrect and missed word for human comprehension of the audio recording.
10. A computing system comprising processing circuitry configured to:
automatically generate, for each of a plurality of audio recordings, a machine-generated transcript of the audio recording;
determine, for each of the audio recordings, an error rate for each the machine-generated transcript, wherein each error rate is generated by comparing the machine-generated transcript to a reference transcript of the audio recording;
receive, for each of the machine-generated transcripts, a subjective ranking selected from a plurality of subjective rank categories;
determine, based on the error rates and subjective rankings for the plurality of audio recordings, objective rank categories defined by error-rate ranges; and
assign to a new machine-generated transcript of a new audio recording not included in the plurality of audio recordings, an objective ranking based on the objective rank categories and an error rate of the new machine-generated transcript.
11. The computing system of claim 10 , wherein the subjective rank categories comprise three categories associated with progressively good transcript quality.
12. The computing system of claim 10 , wherein each of the objective rank categories corresponds to one of the subjective rank categories.
13. The computing system of claim 10 , wherein to determine the objective rank categories, the computing system is further configured to select error-rate boundaries between any two consecutive, non-overlapping subjective rank categories based on a difference between each error rate and the error-rate boundary.
14. The computing system of claim 10 , wherein to determine the objective rank categories, the computing system is further configured to select error-rate boundaries between any two consecutive, overlapping subjective rank categories based on the difference between each error rate and the error-rate boundary, and based on a relative proportion of error rates sharing a common subjective rank category and falling on a common side of the boundary.
15. The computing system of claim 10 , wherein to determine the objective rank categories, the computing system is further configured to apply a support vector machine (SVM) algorithm to the error rates and to the subjective rankings of the machine-generated transcripts in order to select error-rate boundaries between the error-rate ranges.
16. The computing system of claim 10 , wherein the computing system is further configured to determine, based on the objective rankings of a plurality of new machine-generated transcripts not included in the machine-generated transcripts, a relative accuracy of an audio-transcription application configured to generate the plurality of new machine-generated transcripts.
17. The computing system of claim 10 , wherein to determine the error rate for each machine-generated transcript, the computing system is further configured to determine the error rate based on a percent of incorrect and missed words in the machine-generated transcript relative to the reference transcript.
18. The computing system of claim 17 , wherein to determine the error rate for each machine-generated transcript, the computing system is further configured to determine the error rate based on a weighting indicating the relative importance of each incorrect and missed word for human comprehension of the audio recording.
19. A non-transitory computer readable-medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to:
automatically generate, for each of a plurality of audio recordings, a machine-generated transcript of the audio recording;
determine, for each of the audio recordings, an error rate for each the machine-generated transcript, wherein each error rate is generated by comparing the machine-generated transcript to a reference transcript of the audio recording;
receive, for each of the machine-generated transcripts, a subjective ranking selected from a plurality of subjective rank categories;
determine, based on the error rates and subjective rankings for the plurality of audio recordings, objective rank categories defined by error-rate ranges; and
assign, to a new machine-generated transcript of a new audio recording not included in the plurality of audio recordings, an objective ranking based on the objective rank categories and an error rate of the new machine-generated transcript.
20. The computer-readable medium of claim 19 , wherein to determine the objective rank categories, the instructions further configure the processing circuitry to apply a support vector machine (SVM) algorithm to the error rates and to the subjective rankings of the machine-generated transcripts in order to select error-rate boundaries between the error-rate ranges.