Music-aware speaker diarization for transcripts and text-based video editing
Embodiments of the present invention provide systems, methods, and computer storage media for music-aware speaker diarization. In an example embodiment, one or more audio classifiers detect speech and music independently of each other, which facilitates detecting regions in an audio track that contain music but do not contain speech. These music-only regions are compared to the transcript, and any transcription and speakers that overlap in time with the music-only regions are removed from the transcript. In some embodiments, rather than having the transcript display the text from this detected music, a visual representation of the audio waveform is included in the corresponding regions of the transcript.
1. One or more computer storage media storing computer-useable instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising:
detecting, using one or more audio classifiers, (i) speech regions of an audio track that contain detected speech and (ii) music regions of the audio track that contain detected music;
detecting music-only regions based at least on comparing times of the music regions to times of the speech regions;
identifying transcribed singing in a transcript of the audio track based on corresponding sentences overlapping with the music-only regions; and
causing presentation of a visualization of the transcript that omits the transcribed singing from transcript text of the transcript.
2. The one or more computer storage media of claim 1 , wherein detecting the speech regions comprises generating a detection curve representing likelihood over time that speech is present in the audio track, and detecting start and stop times for the speech regions by applying smoothing and thresholding to the detection curve.
3. The one or more computer storage media of claim 1 , the operations further comprising merging adjacent speech regions of the speech regions, or merging adjacent music regions of the music regions, within a threshold temporal spacing of each other.
4. The one or more computer storage media of claim 1 , the operations further comprising removing a set of the music-only regions that are shorter than a threshold duration.
5. The one or more computer storage media of claim 1 ,
wherein the visualization of the transcript includes a visual representation of an audio waveform of the transcribed singing in the transcript.
6. The one or more computer storage media of claim 1 ,
wherein the visualization of the transcript includes a soundbar of the transcribed singing in a row of the transcript without any of the transcript text.
7. The one or more computer storage media of claim 1 ,
wherein the visualization of the transcript includes a spatially condensed visual representation of an audio waveform of the transcribed singing.
8. The one or more computer storage media of claim 1 ,
wherein the visualization of the transcript includes a visual representation of an audio waveform of the transcribed singing annotated with a label identifying a classification of the audio waveform.
9. A method comprising:
generating a representation of speech events identifying regions of an audio track that contain detected speech, and music events identifying regions of the audio track that contain detected music;
detecting music-only events identifying regions of the audio track where the music events do not overlap the speech events;
identifying transcribed singing in a transcript of the audio track based on corresponding sentences overlapping with the music-only events; and
causing presentation of a visualization of the transcript that omits the transcribed singing from transcript text of the transcript.
10. The method of claim 9 , wherein detecting the speech events comprises generating a detection curve representing likelihood over time that speech is present in the audio track, and detecting start and stop times for the speech events by applying smoothing and thresholding to the detection curve.
11. The method of claim 9 , further comprising merging adjacent speech events of the speech events, or merging adjacent music events of the music events, within a threshold temporal spacing of each other.
12. The method of claim 9 , further comprising removing a set of the music-only events that are shorter than a threshold duration.
13. The method of claim 9 , wherein the visualization of the transcript includes a visual representation of an audio waveform of the transcribed singing in the transcript.
14. The method of claim 9 , wherein the visualization of the transcript includes a soundbar of the transcribed singing in a row of the transcript without any of the transcript text.
15. The method of claim 9 , wherein the visualization of the transcript includes a spatially condensed visual representation of an audio waveform of the transcribed singing.
16. The method of claim 9 , wherein the visualization of the transcript includes a visual representation of an audio waveform of the transcribed singing annotated with a label identifying a classification of the audio waveform.
17. A computer system comprising one or more processors and memory configured to provide computer program instructions to the one or more processors, the computer program instructions comprising:
a video interaction engine configured to trigger detecting music-only events that identify regions of an audio track of a video where detected music events do not overlap detected speech events; and
a transcript tool configured to cause presentation of a visualization of a transcript that omits, from transcript text of the transcript, transcribed sentences that overlap with the music-only events.
18. The computer system of claim 17 , wherein the visualization of the transcript includes a visual representation of an audio waveform of the transcribed sentences in the transcript.
19. The computer system of claim 17 , wherein the visualization of the transcript includes a soundbar of the transcribed sentences in a row of the transcript without any of the transcript text.
20. The computer system of claim 17 , wherein the visualization of the transcript includes a spatially condensed visual representation of an audio waveform of the transcribed sentences.