Accuracy of streaming RNN transducer

1. A computer-implemented method for model training, comprising:

training a second end-to-end neural speech recognition model that has a bidirectional encoder to output same symbols from an output probability lattice of the second end-to-end neural speech recognition model as from an output probability lattice of a trained first end-to-end neural speech recognition model haying a unidirectional encoder; and

building a third end-to-end neural speech recognition model that has a unidirectional encoder by training the third end-to-end speech recognition model as a student by using the trained second end-to-end neural speech recognition model as a teacher in a knowledge distillation method, the first and third end-to-end neural speech recognition models being configured to stream output data from a recurrent neural network transducer (RNN-T), and the second end-to-end neural speech recognition model provides the output data offline.

2. The computer-implemented method of claim 1 , wherein training the second end-to-end speech recognition model includes:

obtaining a 1-best symbol for each of nodes of the output probability lattice of the first end-to-end neural speech recognition model by feeding training audio data to the first reference end-to-end speech recognition model, selecting the 1-best symbol from each node in the output posterior lattice, and outputting the 1 best symbol for each node from the unidirectional encoder for training the bidirectional encoder; and

minimizing an end-to-end neural speech recognition model loss and also a cross entropy loss regarding the 1-best symbol.

3. The computer-implemented method of claim 2 , wherein a weighted summation of the cross entropy loss and the end-to-end speech recognition model loss is minimized.

4. The computer-implemented method of claim 2 , wherein training the third end-to-end neural speech recognition model includes:

masking less probable nodes of the output probability lattice of the second end-to-end neural speech recognition model with respect to a minimum probability threshold; and

jointly minimizing (i) the end-to-end neural speech recognition model loss and (ii) a relative divergence over unmasked portions of the output probability lattice of the second end-to-end neural speech recognition model to an output probability lattice of the third end-to-end speech recognition model.

5. The computer-implemented method of claim 4 , wherein the relative divergence is a Kullback-Leibler divergence.

6. The computer-implemented method of claim 1 , wherein training the second end-to-end speech recognition model further includes inputting training audio data into the trained first end-to-end neural speech recognition model to obtain the output probability lattice of the second end-to-end neural speech recognition model.

7. The computer-implemented method of claim 1 , wherein each of the nodes of the output probability lattice of the second end-to-end neural speech recognition. model represent a softmax operation.

8. The computer-implemented method of claim 1 , further comprising performing speech recognition by performing a beam search over an output probability lattice of the trained third end-to-end neural speech recognition model.

9. The computer-implemented method of claim 1 , wherein the second end-to-end neural speech recognition model comprises a more complex neural network than the first end-to-end neural speech recognition model.

10. The computer-implemented method of claim 1 , thither comprising evaluating the output probability lattice of the third end-to-end neural speech recognition model based on a similarity to the output probability lattice of the second end-to-end speech recognition system.

11. The computer-implemented method of claim 1 , wherein the third end- to-end neural speech recognition system is trained until the similarity between the output probability lattices of the third and second end-to-end neural speech recognition models is within a particular similarity range.

12. The computer-implemented method of claim 1 , wherein the method is performed by a speech recognition system.

13. The computer-implemented method of claim 1 , wherein at least one of the first, second and third neural speech recognition models comprise a Recurrent Neural Network Transducer model.

14. The computer-implemented method of claim 1 , wherein the second end-to-end neural speech recognition model comprises an ensemble of end-to-end neural speech recognition models, and the method further comprises selecting a best teacher model from the ensemble responsive to an overlap of search paths through an output probability lattice of each of the end-to-end neural speech recognition models in the ensemble.

15. A computer program product for model training, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:

training a second end-to-end neural speech recognition model that has a bidirectional encoder to output same symbols from an output probability lattice of the second end-to-end neural speech recognition model as from an output probability lattice of a trained first end-to-end neural speech recognition model having, a unidirectional encoder; and

building a third end-to-end neural speech recognition model that has a unidirectional encoder by training the third end-to-end speech recognition model as a student by using the trained second end-to-end neural speech recognition model as a teacher in a knowledge distillation method, the first and third end-to-end neural speech recognition models being configured to stream output data from a recurrent neural network transducer (RNN-T), and the second end-to-end neural speech recognition model provides the output data offline.

16. The computer-implemented method of claim 15 , wherein training the second end-to-end speech recognition model includes:

obtaining a 1-best symbol for each of nodes of the output probability lattice of the first end-to-end neural speech recognition model by feeding training audio data to the first reference end-to-end speech recognition model, selecting the 1-best symbol from each node in the output posterior lattice, and outputting the 1 best symbol for each node from the unidirectional encoder for training the bidirectional encoder; and

minimizing an end-to-end neural speech recognition model loss and also a cross entropy loss regarding the 1-best symbol.

17. The computer-implemented method of claim 16 , wherein a weighted summation of the cross entropy loss and the end-to-end speech recognition model loss is minimized.

18. The computer-implemented method of claim 16 , wherein training the third end-to-end neural speech recognition model includes:

masking less probable nodes of the output probability lattice of the second end-to-end neural speech recognition model with respect to a minimum probability threshold; and

jointly minimizing (i) the end-to-end neural speech recognition model loss and (ii) a relative divergence over unmasked portions of the output probability lattice of the second end-to-end neural speech recognition model to an output probability lattice of the third end-to-end speech recognition model.

19. A computer processing system for model training, comprising:

a memory device for storing program code; and

a hardware processor operatively coupled to the memory device for running the program code to;

train a second end-to-end neural speech recognition model that has a bidirectional encoder to output same symbols from an output probability lattice of the second end-to-end neural speech recognition model as from an output probability lattice of a trained first end-to-end neural speech recognition model having a unidirectional encoder; and

build a third end-to-end neural speech recognition model that has a unidirectional encoder by training the third end-to-end speech recognition model as a student by using the trained second end-to-end neural speech recognition model as a teacher in a knowledge distillation method, the first and third end-to-end neural speech recognition models being configured to stream output data from a recurrent neural network transducer (RNN-T), and the second end-to-end neural speech recognition model provides the output data offline.