Emitting word timings with end-to-end models
A method includes receiving a training example that includes audio data representing a spoken utterance and a ground truth transcription. For each word in the spoken utterance, the method also includes inserting a placeholder symbol before the respective word identifying a respective ground truth alignment for a beginning and an end of the respective word, determining a beginning word piece and an ending word piece, and generating a first constrained alignment for the beginning word piece and a second constrained alignment for the ending word piece. The first constrained alignment is aligned with the ground truth alignment for the beginning of the respective word and the second constrained alignment is aligned with the ground truth alignment for the ending of the respective word. The method also includes constraining an attention head of a second pass decoder by applying the first and second constrained alignments.
1. A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:
receiving audio data characterizing a spoken utterance;
encoding, using an encoder of a neural network speech recognition model, the audio data into a sequence of audio encodings,
processing, using a first decoder of the neural network speech recognition model, the sequence of audio encodings to generate, as output from the first decoder, a plurality of speech recognition hypotheses for the spoken utterance, each speech recognition hypothesis of the plurality of speech recognition hypotheses comprising a corresponding sequence of words;
for each respective speech recognition hypothesis in the top-K speech recognition hypotheses among the plurality of speech recognition hypotheses generated as output from the first decoder, processing, using a second decoder of the neural network speech recognition model, the corresponding sequence of words of the respective speech recognition hypothesis to compute a respective score;
selecting, using the second decoder, as a transcription for the spoken utterance, the corresponding sequence of words of the respective speech recognition hypothesis having the highest respective score; and
determining, using a constrained attention head of the neural network speech recognition model, actual word timings for each word in the corresponding sequence of words selected as the transcription for the spoken utterance.
2. The computer-implemented method of claim 1 , wherein the first decoder comprises a prediction network and a joint network.
3. The computer-implemented method of claim 1 , wherein:
the first decoder of the neural network speech recognition model comprises a recurrent neural network-transducer (RNN-T) decoder; and
the second decoder of the neural network speech recognition model comprises a Listen, Attend, and Spell (LAS) decoder.
4. The computer-implemented method of claim 1 , wherein the data processing hardware resides on a user device that captured the spoken utterance in streaming audio.
5. The computer-implemented method of claim 1 , wherein the operations further comprise displaying, on a screen in communication with the data processing hardware, the transcription of the spoken utterance, the transcription annotated with the actual word timings determined for each word in the corresponding sequence of words selected as the transcription for the spoken utterance.
6. The computer-implemented method of claim 1 , wherein the second decoder comprises a plurality of attention heads.
7. The computer-implemented method of claim 1 , wherein determining the actual word timings for each word in the corresponding sequence of words selected as the transcription for the spoken utterance comprises:
determining a time corresponding to a maximum probability at the constrained attention head of the neural network speech recognition model; and
generating a word start time or a word end time for the determined time corresponding to a maximum probability at the constrained attention head of the neural network speech recognition model.
8. The computer-implemented method of claim 1 , wherein the constrained attention head of the neural network speech recognition model is configured to generate an attention probability that indicates the actual word timings.
9. The computer-implemented method of claim 1 , wherein a training process constrains the constrained attention head of the neural network speech recognition model to generate an attention probability that indicates word timings of an output of the neural network speech recognition model.
10. The computer-implemented method of claim 1 , wherein the plurality of speech recognition hypotheses generated as output from the first decoder comprise a plurality of streaming speech recognition hypotheses.
11. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving audio data characterizing a spoken utterance;
encoding, using an encoder of a neural network speech recognition model, the audio data into a sequence of audio encodings,
processing, using a first decoder of the neural network speech recognition model, the sequence of audio encodings to generate, as output from the first decoder, a plurality of speech recognition hypotheses for the spoken utterance, each speech recognition hypothesis of the plurality of speech recognition hypotheses comprising a corresponding sequence of words;
for each respective speech recognition hypothesis in the top-K speech recognition hypotheses among the plurality of speech recognition hypotheses generated as output from the first decoder, processing, using a second decoder of the neural network speech recognition model, the corresponding sequence of words of the respective speech recognition hypothesis to compute a respective score;
selecting, using the second decoder, as a transcription for the spoken utterance, the corresponding sequence of words of the respective speech recognition hypothesis having the highest respective score; and
determining, using a constrained attention head of the neural network speech recognition model, actual word timings for each word in the corresponding sequence of words selected as the transcription for the spoken utterance.
12. The system of claim 11 , wherein the first decoder comprises a prediction network and a joint network.
13. The system of claim 11 , wherein:
the first decoder of the neural network speech recognition model comprises a recurrent neural network-transducer (RNN-T) decoder; and
the second decoder of the neural network speech recognition model comprises a Listen, Attend, and Spell (LAS) decoder.
14. The system of claim 11 , wherein the data processing hardware resides on a user device that captured the spoken utterance in streaming audio.
15. The system of claim 11 , wherein the operations further comprise displaying, on a screen in communication with the data processing hardware, the transcription of the spoken utterance, the transcription annotated with the actual word timings determined for each word in the corresponding sequence of words selected as the transcription for the spoken utterance.
16. The system of claim 11 , wherein the second decoder comprises a plurality of attention heads.
17. The system of claim 11 , wherein determining the actual word timings for each word in the corresponding sequence of words selected as the transcription for the spoken utterance comprises:
determining a time corresponding to a maximum probability at the constrained attention head of the neural network speech recognition model; and
generating a word start time or a word end time for the determined time corresponding to a maximum probability at the constrained attention head of the neural network speech recognition model.
18. The system of claim 11 , wherein the constrained attention head of the neural network speech recognition model is configured to generate an attention probability that indicates the actual word timings.
19. The system of claim 11 , wherein a training process constrains the constrained attention head of the neural network speech recognition model to generate an attention probability that indicates word timings of an output of the neural network speech recognition model.
20. The system of claim 11 , wherein the plurality of speech recognition hypotheses generated as output from the first decoder comprise a plurality of streaming speech recognition hypotheses.