Emitting word timings with end-to-end models

1. A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising:

receiving a training example for a second pass decoder of a two-pass neural network model, the training example comprising audio data representing a spoken utterance of one or more words and a corresponding ground truth transcription of the spoken utterance;

for each respective word of the one or more words in the spoken utterance:

inserting a placeholder symbol before the respective word;

identifying a respective ground truth alignment for a beginning of the respective word and an end of the respective word;

determining a beginning word piece of the respective word and an ending word piece of the respective word; and

generating a first constrained alignment for the beginning word piece of the respective word and a second constrained alignment for the ending word piece of the respective word, the first constrained alignment aligned with the ground truth alignment for the beginning of the respective word, the second constrained alignment aligned with the ground truth alignment for the ending of the respective word; and

constraining an attention head of the second pass decoder of the two-pass neural network model by applying the training example comprising all of the first constrained alignments and the second constrained alignments for each word of the training example.

2. The computer-implemented method of claim 1 , wherein the beginning word piece and the ending word piece comprise a same word piece for the respective word.

3. The computer-implemented method of claim 1 , wherein the second pass decoder comprises a plurality of attention heads.

4. The computer-implemented method of claim 1 , wherein constraining the attention head comprises constraining an attention probability derived from the attention head of the second pass decoder.

5. The computer-implemented method of claim 1 , wherein the operations further comprise, while training the second pass decoder on the training example:

identifying an expected attention probability for portions of the training example;

determining that the constrained attention head generates an attention probability for at least one of the portions of the training example that fails to match the expected attention probability; and

applying a training penalty to the constrained attention head.

6. The computer-implemented method of claim 5 , wherein the attention probability for the at least one of the portions of the training example occurs at a time corresponding to either the first constrained alignment or the second constrained alignment.

7. The computer-implemented method of claim 5 , wherein the attention probability for the at least one of the portions of the training example occurs at a time corresponding to neither the first constrained alignment nor the second constrained alignment.

8. The computer-implemented method of claim 1 , wherein the operations further comprise, while training the second pass decoder on the training example:

determining that the constrained attention head generates a non-zero attention probability outside of boundaries corresponding to the first constrained alignment and the second constrained alignment; and

applying a training penalty to the constrained attention head.

9. The computer-implemented method of claim 1 , wherein the operations further comprise, while training the second pass decoder on the training example:

minimizing an attention loss for the constrained attention head; and

minimizing a cross entropy loss for the second pass decoder.

10. The computer-implemented method of claim 1 , wherein each respective constrained alignment comprises a timing buffer about the respective ground truth alignment, the timing buffer constraining each of the first constrained alignment and the second constrained alignment to a time interval comprising a first period of time before the respective ground truth alignment and a second period of time after the respective ground truth alignment.

11. The computer-implemented method of claim 1 , wherein the operations further comprise, during execution of the two-pass neural network, using the second pass decoder trained on the training example:

receiving audio data of an utterance;

determining a time corresponding to a maximum probability at the constrained attention head of the second pass decoder; and

generating a word start time or a word end time for the determined time corresponding to a maximum probability at the constrained attention head of the second pass decoder.

12. A system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:

receiving a training example for a second pass decoder of a two-pass neural network model, the training example comprising audio data representing a spoken utterance of one or more words and a corresponding ground truth transcription of the spoken utterance;

for each respective word of the one or more words of the spoken utterance:

inserting a placeholder symbol before the respective word;

identifying a respective ground truth alignment for a beginning of the respective word and an end of the respective word;

determining a beginning word piece of the respective word and an ending word piece of the respective word; and

generating a first constrained alignment for the beginning word piece of the respective word and a second constrained alignment for the ending word piece of the respective word, the first constrained alignment aligned with the ground truth alignment for the beginning of the respective word, the second constrained alignment aligned with the ground truth alignment for the ending of the respective word; and

constraining an attention head of the LAS decoder of the two-pass neural network model by applying the training example comprising all of the first constrained alignments and the second constrained alignments for each word of the training example.

13. The system of claim 12 , wherein the beginning word piece and the ending word piece comprise a same word piece for the respective word.

14. The system of claim 12 , wherein the second pass decoder comprises a plurality of attention heads.

15. The system of claim 12 , wherein constraining the attention head comprises constraining an attention probability derived from the attention head of the second pass decoder.

16. The system of claim 12 , wherein the operations further comprise, while training the second pass decoder on the training example:

identifying an expected attention probability for portions of the training example;

determining that the constrained attention head generates an attention probability for at least one of the portions of the training example that fails to match the expected attention probability; and

applying a training penalty to the constrained attention head.

17. The system of claim 16 , wherein the attention probability for the at least one of the portions of the training example occurs at a time corresponding to either the first constrained alignment or the second constrained alignment.

18. The system of claim 16 , wherein the attention probability for the at least one of the portions of the training example occurs at a time corresponding to neither the first constrained alignment nor the second constrained alignment.

19. The system of claim 12 , wherein the operations further comprise, while training the second pass decoder on the training example:

determining that the constrained attention head generates a non-zero attention probability outside of boundaries corresponding to the first constrained alignment and the second constrained alignment; and

applying a training penalty to the constrained attention head.

20. The system of claim 12 , wherein the operations further comprise, while training the second pass decoder on the training example:

minimizing an attention loss for the constrained attention head; and

minimizing a cross entropy loss for the second pass decoder.

21. The system of claim 12 , wherein each respective constrained alignment comprises a timing buffer about the respective ground truth alignment, the timing buffer constraining each of the first constrained alignment and the second constrained alignment to a time interval comprising a first period of time before the respective ground truth alignment and a second period of time after the respective ground truth alignment.

22. The system of claim 12 , wherein the operations further comprise, during execution of the two-pass neural network, using the second pass decoder trained on the training example:

receiving audio data of an utterance;

determining a time corresponding to a maximum probability at the constrained attention head of the second pass decoder; and

generating a word start time or a word end time for the determined time corresponding to a maximum probability at the constrained attention head of the second pass decoder.