Automated transcript generation from multi-channel audio
Systems and methods are described for generating a transcript of a legal proceeding or other multi-speaker conversation or performance in real time or near-real time using multi-channel audio capture. Different speakers or participants in a conversation may each be assigned a separate microphone that is placed in proximity to the given speaker, where each audio channel includes audio captured by a different microphone. Filters may be applied to isolate each channel to include speech utterances of a different speaker, and these filtered channels of audio data may then be processed in parallel to generate speech-to-text results that are interleaved to form a generated transcript.
1. A system comprising:
a plurality of microphones;
audio mixer hardware configured to process a plurality of audio channels, wherein each of the plurality of microphones corresponds to a different channel of the plurality of audio channels; and
a computing system in communication with the audio mixer hardware and comprising memory and a processor, the computing system configured with processor-executable instructions to perform operations comprising:
receiving speaker identification information for each of the plurality of audio channels, wherein the speaker identification information for each individual audio channel identifies a person assigned to the individual audio channel;
receiving at least a portion of multi-channel streaming audio from the audio mixer hardware, wherein the multi-channel streaming audio comprises audio signals captured from each of the plurality of microphones on a different channel of the plurality of audio channels, wherein streaming audio of a first channel of the plurality of audio channels includes speech utterances spoken by two or more people;
obtaining a plurality of filtered audio channels that are generated based on the multi-channel streaming audio, wherein each of the filtered audio channels includes speech utterances spoken by a different person assigned to the individual corresponding audio channel;
obtaining text results for each of the filtered audio channels by providing audio data from each of the filtered audio channels to one or more speech-to-text services or modules, wherein the text results for each of the filtered audio channels represent words spoken by a different speaker; and
generating a transcript of at least a portion of multi-speaker conversation based on the text results, wherein the transcript includes identification of a respective speaker each time that a change in speaker occurs in the transcript.
2. The system of claim 1 , wherein the operations further comprise:
selecting a speech model to be used with respect to audio data for each of two or more of the plurality of audio channels, wherein a first speech model selected for a first audio channel is based at least in part on vocal characteristic information of a first person assigned to the first audio channel, wherein the first speech model is used for the automatic speech recognition as applied to audio data from a first filtered audio channel corresponding to the first audio channel.
3. The system of claim 2 , wherein the first speech model is selected for the first audio channel based at least in part on at least one of an accent, tonality or gender of the first person.
4. The system of claim 1 , wherein, for each individual audio channel of the plurality of audio channels, a person is assigned to the individual audio channel based on the person being physically located closer to a microphone assigned to the individual audio channel than to any other microphone of the plurality of microphones.
5. A computer-implemented method comprising:
as implemented by one or more computing devices configured with specific executable instructions,
receiving speaker identification information for each of a plurality of audio channels, wherein the speaker identification information for each individual audio channel identifies a person assigned to the individual audio channel, wherein each of the plurality of audio channels are associated with a different microphone of a plurality of microphones;
receiving multi-channel streaming audio on the plurality of audio channels, wherein the multi-channel streaming audio comprises audio signals captured from each of the plurality of microphones on a different channel of the plurality of audio channels, wherein streaming audio of a first channel of the plurality of audio channels includes speech utterances spoken by two or more people;
obtaining a plurality of filtered audio channels that are generated based on the multi-channel streaming audio, such that each of the filtered audio channels includes speech utterances spoken by a different person assigned to the individual corresponding audio channel;
obtaining text results for each of the filtered audio channels by providing audio from each of the filtered audio channels to one or more speech-to-text services or modules, wherein the text results for each of the filtered audio channels represent words spoken by a different speaker; and
generating a transcript of at least a portion of multi-speaker conversation based on the text results, wherein the transcript includes identification of a respective speaker each time that a change in speaker occurs in the transcript.
6. The computer-implemented method of claim 5 , further comprising:
identifying a redundant word among text results of two or more of the filtered audio channels, wherein the redundant word comprises a word appearing in the text results of each of the two or more channels at matching timestamps;
determining a correct channel for the redundant word based at least in part on a comparison of filtered audio from each of the two or more channels at the matching timestamps; and
removing the redundant word from text results of one or more of the filtered audio channels other than the correct channel.
7. The computer-implemented method of claim 5 , further comprising assembling an audio file that interleaves audio from different filtered audio channels according to which filtered audio channel corresponds to an active speaker at a given point.
8. The computer-implemented method of claim 5 , wherein the multi-channel streaming audio is captured during a legal proceeding, and wherein the transcript is generated in real time during the legal proceeding.
9. The computer-implemented method of claim 8 , further comprising automatically formatting text of the transcript using a template associated with a proceeding type of the legal proceeding.
10. The computer-implemented method of claim 5 , further comprising causing display of at least a portion of the transcript that includes one or more words visually highlighted to indicate a relative confidence level associated with automatic speech recognition of the one or more words.
11. The computer-implemented method of claim 5 , wherein the plurality of filtered audio channels are generated at least in part by applying at least one of beamforming, adaptive weighting or echo cancellation to each of the plurality of audio channels.
12. A non-transitory computer-readable medium having stored thereon executable instructions that direct a computer system to perform operations comprising:
receiving speaker identification information for each of a plurality of audio channels, wherein the speaker identification information for each individual audio channel identifies a person assigned to the individual audio channel, wherein each of the plurality of audio channels are associated with a different microphone of a plurality of microphones;
receiving multi-channel streaming audio on the plurality of audio channels, wherein the multi-channel streaming audio comprises audio signals captured from each of the plurality of microphones on a different channel of the plurality of audio channels, wherein streaming audio of a first channel of the plurality of audio channels includes speech utterances spoken by two or more people;
obtaining a plurality of filtered audio channels that are generated based on the multi-channel streaming audio, such that each of the filtered audio channels includes speech utterances spoken by a different person assigned to the individual corresponding audio channel;
obtaining text results for each of the filtered audio channels by providing audio from each of the filtered audio channels to one or more speech-to-text services or modules, wherein the text results for each of the filtered audio channels represent words spoken by a different speaker; and
generating a transcript of at least a portion of multi-speaker conversation based on the text results, wherein the transcript includes identification of a respective speaker each time that a change in speaker occurs in the transcript.
13. The non-transitory computer-readable medium of claim 12 , the operations further comprising:
identifying a redundant word among text results of two or more of the filtered audio channels, wherein the redundant word comprises a word appearing in the text results of each of the two or more channels at matching timestamps;
determining a correct channel for the redundant word based at least in part on a comparison of filtered audio from each of the two or more channels at the matching timestamps; and
removing the redundant word from text results of one or more of the filtered audio channels other than the correct channel.
14. The non-transitory computer-readable medium of claim 12 , the operations further comprising assembling an audio file that interleaves audio from different filtered audio channels according to which filtered audio channel corresponds to an active speaker at a given point.
15. The non-transitory computer-readable medium of claim 12 , wherein the multi-channel streaming audio is captured during an event, wherein the transcript is generated in real time during the event.
16. The non-transitory computer-readable medium of claim 15 , wherein the event comprises a television broadcast or a live performance.
17. The non-transitory computer-readable medium of claim 15 , wherein the event is a deposition in a legal proceeding.
18. The non-transitory computer-readable medium of claim 12 , the operations further comprising causing display of at least a portion of the transcript that includes one or more words visually highlighted to indicate a relative confidence level associated with automatic speech recognition of the one or more words.
19. The non-transitory computer-readable medium of claim 12 , the operations further comprising generating a synchronized audiovisual presentation that presents portions of the transcript in synchronization with corresponding portions of video, wherein the synchronized audiovisual presentation is generated based at least in part on timestamps associated with the video that correspond to a clock that was also used in timestamping corresponding audio captured from the plurality of microphones.