Human resolution of repeated phrases in a hybrid transcription system
When transcribing audio recordings, such as legal depositions, phrases may be repeated throughout the recordings, but these repeated phrases get transcribed incorrectly by an automatic speech recognition (ASR) system. In order to assist a transcriber to correctly resolve such phrases, some embodiments described herein involve a computer that receives an audio recording that includes speech, generates a transcription of the audio recording utilizing an ASR system, and clusters segments of the audio recording into clusters of similar utterances. The computer provides a transcriber with certain segments of the audio recording, which include similar utterances belonging to a certain cluster, along with transcriptions of the certain segments. The computer receives from the transcriber: an indication of which of the certain segments include repetitions of a phrase, and a correct transcription of the phrase. The computer then updates the transcription of the audio recording based on the indication and the correct transcription.
1. A system configured to assist in transcription of a repeated phrase, comprising:
a frontend server configured to transmit an audio recording comprising speech of first and second people; and
a backend server configured to:
generate a transcription of the audio recording utilizing an automatic speech recognition (ASR) system;
cluster segments of the audio recording into clusters of similar utterances;
select first and second segments of the audio recording that comprise similar utterances spoken by the first and second people, respectively;
generate, for each transcriber from among transcribers, feature values based on the first and second segments and utilize a machine learning-based model to calculate, based on the feature values, expected accuracies of transcriptions of the first and second segments were they transcribed by the transcriber; wherein the machine learning-based model is generated based on training data comprising additional feature values generated based on additional segments of additional audio recordings, and values of accuracies of transcriptions, by the transcriber, of the additional segments;
select a certain transcriber, from among the transcribers, whose expected accuracies reach a predetermined threshold;
provide the certain transcriber with the first and second segments of the audio recording and with transcriptions of the first and second segments;
receive from the certain transcriber: an indication indicating whether the first and second segments comprise repetitions of a phrase, and a correct transcription of said phrase; and
update the transcription of the audio recording based on the indication and the correct transcription.
2. The system of claim 1 , wherein the backend server is further configured to utilize the indication to update a phonetic model utilized by the ASR system to reflect one or more pronunciations of the phrase.
3. The system of claim 1 , wherein the backend server is further configured to update a language model utilized by the ASR system to include the correct transcription of the phrase.
4. The system of claim 1 , wherein the backend server is further configured to cluster the segments utilizing dynamic time warping (DTW) of acoustic feature representations of the segments.
5. The system of claim 1 , wherein the backend server is further configured cluster the segments based on similarity of paths corresponding to the segments in a lattice constructed by the ASR system.
6. The system of claim 1 , wherein the backend server is further configured to represent each segment of audio and a product of ASR of the segment using a vector of feature values that comprises: one or more feature values indicative of acoustic properties of the segment, and at least some feature values indicative of phonetic transcription properties calculated by the ASR system; the backend server is further configured to utilize a distance function that operates on pairs of vectors of feature values.
7. The system of claim 1 , wherein the backend server is further configured to update the transcription of the audio recording responsive to the indication indicating that a number of the segments that comprise an utterance of the phrase is greater than a threshold that is at least two.
8. The system of claim 1 , wherein the expected accuracies of the transcriptions of the first and second segments were they transcribed by the certain transcriber are indicative of expected word error rates (WER) in said transcriptions of the first and second segments.
9. A method for assisting in transcription of a repeated phrase, comprising:
receiving an audio recording comprising speech of first and second people;
generating a transcription of the audio recording utilizing an automatic speech recognition (ASR) system;
clustering segments of the audio recording into clusters of similar utterances;
selecting first and second segments of the audio recording that comprise similar utterances spoken by the first and second people, respectively;
generating, for each transcriber from among transcribers, feature values based on the first and second segments and utilizing a machine learning-based model to calculate, based on the feature values, expected accuracies of transcriptions of the first and second segments were they transcribed by the transcriber; wherein the machine learning-based model is generated based on training data comprising additional feature values generated based on additional segments of additional audio recordings, and values of accuracies of transcriptions, by the transcriber, of the additional segments;
selecting a certain transcriber, from among the transcribers, whose expected accuracies reach a predetermined threshold;
providing the certain transcriber with the first and second segments of the audio recording and with transcriptions of the first and second segments;
receiving from the certain transcriber: an indication indicating whether the first and second segments comprise repetitions of a phrase, and a correct transcription of said phrase; and
updating the transcription of the audio recording based on the indication and the correct transcription.
10. The method of claim 9 , further comprising utilizing the indication to update a phonetic model utilized by the ASR system to reflect one or more pronunciations of the phrase.
11. The method of claim 10 , wherein the expected accuracies of the transcriptions of the first and second segments were they transcribed by the certain transcriber are indicative of expected word error rates (WER) in said transcriptions of the first and second segments.
12. The method of claim 9 , further comprising updating a language model utilized by the ASR system to include the correct transcription of the phrase.
13. The method of claim 9 , further comprising clustering the segments utilizing dynamic time warping (DTW) of acoustic feature representations of the segments.
14. The method of claim 9 , further comprising clustering the segments based on similarity of paths corresponding to the segments in a lattice constructed by the ASR system.
15. The method of claim 9 , further comprising updating the transcription of the audio recording based on the indication indicating that a number of the segments that comprise an utterance of the phrase is greater than a threshold that is at least two.
16. A non-transitory computer-readable medium having instructions stored thereon that, in response to execution by a system including a processor and memory, causes the system to perform operations comprising:
receiving an audio recording comprising speech of first and second people;
generating a transcription of the audio recording utilizing an automatic speech recognition (ASR) system;
clustering segments of the audio recording into clusters of similar utterances;
selecting first and second segments of the audio recording that comprise similar utterances spoken by the first and second people, respectively;
generating, for each transcriber from among transcribers, feature values based on the first and second segments and utilizing a machine learning-based model to calculate, based on the feature values, expected accuracies of transcriptions of the first and second segments were they transcribed by the transcriber; wherein the machine learning-based model is generated based on training data comprising additional feature values generated based on additional segments of additional audio recordings, and values of accuracies of transcriptions, by the transcriber, of the additional segments;
selecting a certain transcriber, from among the transcribers, whose expected accuracies reach a predetermined threshold;
providing the certain transcriber with the first and second segments of the audio recording and with transcriptions of the first and second segments;
receiving from the certain transcriber: an indication indicating whether the first and second segments comprise repetitions of a phrase, and a correct transcription of said phrase; and
updating the transcription of the audio recording based on the indication and the correct transcription.
17. The non-transitory computer-readable medium of claim 16 , further comprising instructions defining a step of utilizing the indication to update a phonetic model utilized by the ASR system to reflect one or more pronunciations of the phrase.
18. The non-transitory computer-readable medium of claim 16 , further comprising instructions defining a step of updating a language model utilized by the ASR system to include the correct transcription of the phrase.
19. The non-transitory computer-readable medium of claim 16 , wherein the expected accuracies of the transcriptions of the first and second segments were they transcribed by the certain transcriber are indicative of expected word error rates (WER) in said transcriptions of the first and second segments.
20. The non-transitory computer-readable medium of claim 16 , further comprising instructions defining a step of clustering the segments utilizing dynamic time warping (DTW) of acoustic feature representations of the segments.