Pre-training with alignments for recurrent neural network transducer based end-to-end speech recognition
Techniques performed by a data processing system for training a Recurrent Neural Network Transducer (RNN-T) herein include encoder pretraining by training a neural network-based token classification model using first token-aligned training data representing a plurality of utterances, where each utterance is associated with a plurality of frames of audio data and tokens representing each utterance are aligned with frame boundaries of the plurality of audio frames; obtaining first cross-entropy (CE) criterion from the token classification model, wherein the CE criterion represent a divergence between expected outputs and reference outputs of the model; pretraining an encoder of an RNN-T based on the first CE criterion; and training the RNN-T with second training data after pretraining the encoder of the RNN-T. These techniques also include whole-network pre-training of the RNN-T. A RNN-T pretrained using these techniques may be used to process audio data that includes spoken content to obtain a textual representation.
1. A data processing system comprising:
a processor; and
a computer-readable medium storing executable instructions for causing the processor to perform operations of:
receiving an audio input comprising spoken content;
analyzing the audio input using a Recurrent Neural Network-Transducer (RNN-T) to obtain a first textual output representing the spoken content, the RNN-T being pretrained using whole network pretraining, wherein the whole-network pretraining pretrains the RNN-T as whole using a cross-entropy (CE) criterion by pretraining an encoder of the RNN-T using a two-dimensional label matrix for each utterance included in training data to train an encoder of the RNN-T and by pretraining a prediction network of the RNN-T using a three-dimensional label matrix derived from the two-dimensional label matrix, wherein the CE criterion represents a divergence between expected outputs and reference outputs of a model; and
processing the first textual output with an application on the data processing system.
2. The data processing system of claim 1 , wherein for the encoder pretraining, the encoder is trained using the CE criterion from a neural network-based token classification model, and the neural network-based token classification model is trained using first token-aligned training data representing a plurality of utterances, wherein each utterance is associated with a plurality of frames of audio data, and wherein tokens representing each utterance are aligned with frame boundaries of the plurality of frames of audio data.
3. A data processing system comprising:
a processor; and
a computer-readable medium storing executable instructions for causing the processor to perform operations of:
training a neural network-based token classification model using first token-aligned training data representing a plurality of utterances, wherein each utterance is associated with a plurality of frames of audio data, and wherein tokens representing each utterance are aligned with frame boundaries of the plurality of frames of audio data;
obtaining a first cross-entropy (CE) criterion from the neural network-based token classification model, wherein the CE criterion represents a divergence between expected outputs and reference outputs of the model;
pretraining an encoder of an RNN-Transducer (RNN-T) based on the first CE criterion obtained from the neural network-based token classification model using a two-dimensional label matrix for each utterance included in the training data;
determining a three-dimensional label matrix for each respective utterance from the two-dimensional label matrix for the respective utterance, the three-dimensional label matrix comprising an alignment path that represents a series of tokens of the respective utterance over a series of frames of audio data;
pretraining a prediction network of the RNN-T based on the CE criterion obtained from the neural network-based token classification model using the three-dimensional label matrix; and
training the RNN-T with second training data after pretraining the encoder of the RNN-T.
4. The data processing system of claim 3 , wherein the computer-readable medium further stores executable instructions for causing the processor to perform a function of generating the token-aligned training data from a source training data set by:
accessing a source training data element representing an utterance of a word;
dividing the word into a plurality of tokens;
determining a number of frames of audio data comprising the utterance of the word;
adding the source training data element to the token-aligned training data responsive to the number of frames comprising the utterance exceeding or matching a number of tokens comprising the plurality of tokens.
5. The data processing system of claim 4 , wherein the computer-readable medium further stores executable instructions for causing the processor to perform a function of discarding the source training data element responsive to the number of tokens exceeding the number of frames comprising the utterance.
6. The data processing system of claim 4 , wherein to train the RNN-T with second training data, the computer-readable medium further stores executable instructions for causing the processor to perform a function of processing an output of the encoder of the RNN-T with an additional fully-connected layer to add alignment information to a high level representation of each frame of acoustic data.
7. The data processing system of claim 3 , wherein the computer-readable medium further stores executable instructions for causing the processor to perform a function of pretraining the prediction network of the RNN-T based on the CE criterion obtained from the neural network-based token classification model.
8. The data processing system of claim 3 , wherein determining the three-dimensional label matrix further comprises:
identifying a space in the alignment path representing a pause in the utterance; and
setting a blank token in the alignment path at a location of the space.
9. The data processing system of claim 8 , wherein the computer-readable medium further stores executable instructions for causing the processor to perform a function of optimizing a CE loss between the three-dimensional label matrix and an output of the model.
10. A data processing system comprising:
a processor; and
a computer-readable medium storing executable instructions for causing the processor to perform operations of:
training a neural network-based token classification model using first token-aligned training data representing a plurality of utterances, wherein each utterance is associated with a plurality of frames of audio data, and wherein tokens representing each utterance are aligned with frame boundaries of the plurality of frames of audio data;
obtaining a first cross-entropy (CE) criterion from the neural network-based token classification model, wherein the CE criterion represents a divergence between expected outputs and reference outputs of the model;
pretraining an encoder of an RNN-Transducer (RNN-T) based on the first CE criterion obtained from the neural network-based token classification model; and
training the RNN-T with second training data after pretraining the encoder of the RNN-T, wherein training the RNN-T with the second training data includes:
processing an output of the encoder of the RNN-T with an additional fully-connected layer to add alignment information to a high level representation of each frame of acoustic data, wherein the output of the fully-connected layer is calculated by:
L
enc
=
∑
k
=
1
K
y
t
,
k
*
log
(
softmax
(
f
fc
(
h
t
,
k
enc
)
)
)
where ƒ fc represents a fully-connected layer, k is a label index and K denotes a largest dimension of an output of a joint network of the RNN-T, and y t is a word piece label for each input frame x t .
11. A method performed by a data processing system for training a Recurrent Neural Network-Transducer (RNN-T), the method comprising:
training a neural network-based token classification model using first token-aligned training data representing a plurality of utterances, wherein each utterance is associated with a plurality of frames of audio data, and wherein tokens representing each utterance are aligned with frame boundaries of the plurality of frames of audio data;
obtaining a first cross-entropy (CE) criterion from the neural network-based token classification model, wherein the CE criterion represents a divergence between expected outputs and reference outputs of the model;
pretraining an encoder of an RNN-Transducer (RNN-T) based on the first CE criterion obtained from the neural network-based token classification model using a two-dimensional label matrix for each utterance included in the training data;
determining a three-dimensional label matrix for each respective utterance from the two-dimensional label matrix for the respective utterance, the three-dimensional label matrix comprising an alignment path that represents a series of tokens of the respective utterance over a series of frames of audio data;
pretraining a prediction network of the RNN-T based on the CE criterion obtained from the neural network-based token classification model using the three-dimensional label matrix; and
training the RNN-T with second training data after pretraining the encoder of the RNN-T.
12. The method of claim 11 , further comprising generating the token-aligned training data from a source training data set by:
accessing a source training data element representing an utterance of a word;
dividing the word into a plurality of tokens;
determining a number of frames of audio data comprising the utterance of the word;
adding the source training data element to the token-aligned training data responsive to the number of frames comprising the utterance exceeding or matching a number of tokens comprising the plurality of tokens.
13. The method of claim 12 , further comprising:
discarding the source training data element responsive to the number of tokens exceeding the number of frames comprising the utterance.
14. The method of claim 12 , wherein training the RNN-T with second training data further comprises:
processing an output of the encoder of the RNN-T with an additional fully-connected layer to add alignment information to a high level representation of each frame of acoustic data.
15. The method of claim 11 , further comprising:
pretraining the prediction network of the RNN-T based on the CE criterion obtained from the neural network-based token classification model.
16. The method of claim 11 , wherein determining the three-dimensional label matrix further comprises:
identifying a space in the alignment path representing a pause in the utterance;
setting a blank token in the alignment path at a location of the space; and
optimizing a CE loss between the three-dimensional label matrix and an output of the model.
17. A method performed by a data processing system for training a Recurrent Neural Network-Transducer (RNN-T), the method comprising:
training a neural network-based token classification model using first token-aligned training data representing a plurality of utterances, wherein each utterance is associated with a plurality of frames of audio data, and wherein tokens representing each utterance are aligned with frame boundaries of the plurality of frames of audio data;
obtaining a first cross-entropy (CE) criterion from the neural network-based token classification model, wherein the CE criterion represents a divergence between expected outputs and reference outputs of the model;
pretraining an encoder of an RNN-Transducer (RNN-T) based on the first CE criterion obtained from the neural network-based token classification model; and
training the RNN-T with second training data after pretraining the encoder of the RNN-T, wherein training the RNN-T with the second training data includes:
processing an output of the encoder of the RNN-T with an additional fully-connected layer to add alignment information to a high level representation of each frame of acoustic data, wherein the output of the fully-connected layer is calculated by:
L
enc
=
∑
k
=
1
K
y
t
,
k
*
log
(
softmax
(
f
fc
(
h
t
,
k
enc
)
)
)
Where ƒ fc represents a fully connected layer, k is a label index and K denotes a largest dimension of an output of a joint network of the RNN-T, and y t is a word piece label for each input frame x t .