End-to-end speech recognition adapted for multi-speaker applications
A system for performing end-to-end automatic speech recognition (ASR). The system configured to collect a sequence of acoustic frames associated with a mixture of speeches performed by multiple speakers. Each frame from the sequence of acoustic frames is encoded using a multi-head encoder which encodes each frame into a likelihood of a transcription output and a likelihood of an identity of a speaker. The multi-head encoder thus produces a sequence of likelihoods of transcription outputs and a sequence of likelihoods of identities of the speakers corresponding to the sequence of acoustic frames that are decoded using a decoder performing an alignment operation for producing a sequence of transcription outputs annotated with identities of the speakers, for performing speaker separation.
1 . An end-to-end automatic speech recognition (ASR) system, comprising: a processor; and a memory having instructions stored thereon, wherein the processor is configured to execute the stored instructions to cause the ASR system to:
collect a sequence of acoustic frames providing a digital representation of an acoustic signal including a mixture of speeches performed by multiple speakers;
encode each frame from the sequence of acoustic frames with a multi-head encoder encoding each frame into a likelihood of a transcription output and a likelihood of an identity of a speaker to produce a sequence of likelihoods of transcription outputs and a sequence of likelihoods of identities of the speakers corresponding to the sequence of acoustic frames;
decode the sequence of likelihoods of transcription outputs and the sequence of likelihoods of identities of the speakers with a decoder performing an alignment producing a sequence of transcription outputs annotated with identities of the speakers, wherein the multi-head encoder and the decoder form at least a part of a neural network trained with an extended connectionist temporal classification (CTC) objective function to enforce an alignment between an input of the neural network and an output of the neural network on a graph with nodes indicative of transcription outputs and edges indicative of speaker transitions; and
submit the sequence of transcription outputs annotated with the identities of the speakers into a downstream application.
2 . The ASR system of claim 1 , wherein the decoder uses a beam search to produce a sequence of chronologically ordered linguistic tokens where each token is associated with a speaker identity.
3 . The ASR system of claim 2 , wherein the beam search is configured to perform an operation including one or a combination of (1) generating a speaker transition probability and a linguistic token probability, (2) computing a score of a linguistic token, (3) expanding a list of prefixes of all speakers from a set of possible speakers, and (4) computing a score of a prefix by considering separately sub-sequences of different speakers.
4 . The ASR system of claim 1 , wherein the encoder comprises an acoustic encoder configured to process the acoustic signal and generate a sequence of encoder states, and the decoder comprises an attention-based decoder.
5 . The ASR system of claim 1 , wherein the neural network is trained to achieve multiple objectives by minimizing a loss function including a first component associated with an error in speech recognition and a second component associated with an error in speaker identification.
6 . The ASR system of claim 1 , wherein the extended CTC objective function is an extended graph based temporal classification (GTC-e) objective function, wherein the GTC-e objective function uses supervisory information from a directed graph of nodes connected by edges representing labels and transitions among the labels, wherein the directed graph represents possible alignment paths for a sequence of probability distributions outputted by the neural network and the labels.
7 . The ASR system of claim 6 , wherein the directed graph represents multiple possible alignment paths for the sequence of probability distributions and a sequence of the labels such that possible passes through a structure of the directed graph allows multiple unique label sequences, which are obtained after collapsing label repetitions and removing blank labels from the multiple unique label sequences, thereby resulting in a non-monotonic alignment between the sequence of the labels and the sequence of probability distributions.
8 . The ASR system of claim 7 , wherein the non-monotonic alignment is encoded in the structure of the directed graph by allowing transitions from one label to multiple other non-blank labels, by allowing transitions from one label to multiple other blank labels, or both.
9 . The ASR system of claim 1 , wherein the extended CTC objective function is a graph based temporal classification-transducer (GTC-T) objective function.
10 . The ASR system of claim 1 , wherein the nodes of the directed graph are indicative of tokens from all speakers in a chronological order.
11 . The ASR system of claim 1 , wherein the edges of the directed graph are indicative of speaker identification information.
12 . A computer-implemented method for end-to-end automatic speech recognition (ASR), comprising:
collecting a sequence of acoustic frames providing a digital representation of an acoustic signal including a mixture of speeches performed by multiple speakers;
encoding each frame from the sequence of acoustic frames with a multi-head encoder encoding each frame into a likelihood of a transcription output and a likelihood of an identity of a speaker to produce a sequence of likelihoods of transcription outputs and a sequence of likelihoods of identities of the speakers corresponding to the sequence of acoustic frames;
decoding the sequence of likelihoods of transcription outputs and the sequence of likelihoods of identities of the speakers with a decoder performing an alignment producing a sequence of transcription outputs annotated with identities of the speakers, wherein the multi-head encoder and the decoder form at least a part of a neural network trained with an extended connectionist temporal classification (CTC) objective function to enforce an alignment between an input of the neural network and an output of the neural network on a graph with nodes indicative of transcription outputs and edges indicative of speaker transitions; and
submitting the sequence of transcription outputs annotated with the identities of the speakers into a downstream application.
13 . The method of claim 12 , wherein the decoder uses a beam search to produce a sequence of chronologically ordered linguistic tokens where each token is associated with a speaker identity.
14 . The method of claim 13 , wherein the beam search is configured to perform an operation including one or a combination of (1) generating a speaker transition probability and a linguistic token probability, (2) computing a score of a linguistic token, (3) expanding a list of prefixes of all speakers from a set of possible speakers, and (4) computing a score of a prefix by considering separately sub-sequences of different speakers.
15 . The method of claim 12 , wherein the encoder comprises a self-attention encoder and the decoder comprises an attention-based decoder.
16 . The method of claim 12 , wherein the neural network is trained to achieve multiple objectives by minimizing a loss function including a first component associated with an error in speech recognition and a second component associated with an error in speaker identification.
17 . The method of claim 12 , wherein the extended CTC objective function is an extended graph based temporal classification (GTC-e) objective function, wherein the GTC-e objective function uses supervisory information from a directed graph of nodes connected by edges representing labels and transitions among the labels, wherein the directed graph represents possible alignment paths for a sequence of probability distributions outputted by the neural network and the labels.
18 . The method of claim 17 , wherein the directed graph represents multiple possible alignment paths for the sequence of probability distributions and a sequence of the labels such that possible passes through a structure of the directed graph allows multiple unique label sequences, which are obtained after collapsing label repetitions and removing blank labels from the multiple unique label sequences, thereby resulting in a non-monotonic alignment between the sequence of the labels and the sequence of probability distributions.
19 . The method of claim 18 , wherein the non-monotonic alignment is encoded in the structure of the directed graph by allowing transitions from one label to multiple other non-blank labels, by allowing transitions from one label to multiple other blank labels, or both.
20 . The method of claim 12 , wherein the extended CTC objective function is a graph based temporal classification-transducer (GTC-T) objective function.