Accuracy of streaming RNN transducer
View Patent ↗A computer-implemented method is provided for model training. The method includes training a second end-to-end neural speech recognition model that has a bidirectional encoder to output same symbols from an output probability lattice of the second end-to-end neural speech recognition model as from an output probability lattice of a trained first end-to-end neural speech recognition model having a unidirectional encoder. The method also includes building a third end-to-end neural speech recognition model that has a unidirectional encoder by training the third end-to-end neural speech recognition model as a student by using the trained second end-to-end neural speech recognition model as a teacher in a knowledge distillation method.
1. A computer-implemented method for model training, comprising:
training a second end-to-end neural speech recognition model that has a bidirectional encoder to output same symbols from an output probability lattice of the second end-to-end neural speech recognition model as from an output probability lattice of a trained first end-to-end neural speech recognition model having a unidirectional encoder that streams the output data from a recurrent neural network (RNN) transducer; and
building a third end-to-end neural speech recognition model configured to stream output data from an RNN transducer and comprising a unidirectional encoder by training the third end-to-end speech recognition model as a student using the trained second end-to-end neural speech recognition model as a teacher in a knowledge distillation method, with output data being provided offline by the second end-to-end neural network speech recognition model.
2. The computer-implemented method of claim 1 , wherein training the second end-to-end speech recognition model includes:
obtaining a 1-best symbol for each of nodes of the output probability lattice of the first end-to-end neural speech recognition model; and
minimizing an end-to-end neural speech recognition model loss and also a cross entropy loss regarding the 1-best symbol.
3. The computer-implemented method of claim 2 , wherein a weighted summation of the cross entropy loss and the end-to-end speech recognition model loss is minimized.
4. The computer-implemented method of claim 2 , wherein training the third end-to-end neural speech recognition model includes:
masking less probable nodes of the output probability lattice of the second end-to-end neural speech recognition model with respect to a minimum probability threshold; and
jointly minimizing (i) the end-to-end neural speech recognition model loss and (ii) a relative divergence over unmasked portions of the output probability lattice of the second end-to-end neural speech recognition model to an output probability lattice of the third end-to-end speech recognition model.
5. The computer-implemented method of claim 4 , wherein the relative divergence is a Kullback-Leibler divergence.
6. The computer-implemented method of claim 1 , wherein training the second end-to-end speech recognition model further includes inputting training audio data into the trained first end-to-end neural speech recognition model to obtain the output probability lattice of the second end-to-end neural speech recognition model.
7. The computer-implemented method of claim 1 , wherein each node of the output probability lattice of the second end-to-end neural speech recognition model represents a softmax operation.
8. The computer-implemented method of claim 1 , further comprising performing speech recognition by performing a beam search over an output probability lattice of the trained third end-to-end neural speech recognition model.
9. The computer-implemented method of claim 1 , wherein the second end-to-end neural speech recognition model comprises a more complex neural network than the first end-to-end neural speech recognition model.
10. The computer-implemented method of claim 1 , further comprising evaluating the output probability lattice of the third end-to-end neural speech recognition model based on a similarity to the output probability lattice of the second end-to-end neural speech recognition model.
11. The computer-implemented method of claim 1 , wherein the third end-to-end neural speech recognition model is trained until a similarity between the output probability lattices of the third and second end-to-end neural speech recognition models is within a particular similarity range.
12. The computer-implemented method of claim 1 , wherein the method is performed by a speech recognition system.
13. The computer-implemented method of claim 1 , wherein at least one of the first, second and third neural speech recognition models comprise a Recurrent Neural Network Transducer model.
14. The computer-implemented method of claim 1 , wherein the second end-to-end neural speech recognition model comprises an ensemble of end-to-end neural speech recognition models, and the method further comprises selecting a best teacher model from the ensemble responsive to an overlap of search paths through an output probability lattice of each of the end-to-end neural speech recognition models in the ensemble.
15. A computer program product for model training, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
training a second end-to-end neural speech recognition model that has a bidirectional encoder to output same symbols from an output probability lattice of the second end-to-end neural speech recognition model as from an output probability lattice of a trained first end-to-end neural speech recognition model having a unidirectional encoder that streams the output data from a recurrent neural network (RNN) transducer; and
building a third end-to-end neural speech recognition model configured to stream output data from an RNN transducer and comprising a unidirectional encoder by training the third end-to-end speech recognition model as a student using the trained second end-to-end neural speech recognition model as a teacher in a knowledge distillation method, with output data being provided offline by the second end-to-end neural network speech recognition model.
16. The computer program product of claim 15 , wherein training the second end-to-end speech recognition model includes:
obtaining a 1-best symbol for each of nodes of the output probability lattice of the first end-to-end neural speech recognition model; and
minimizing an end-to-end neural speech recognition model loss and also a cross entropy loss regarding the 1-best symbol.
17. The computer program product of claim 16 , wherein a weighted summation of the cross-entropy loss and the end-to-end speech recognition model loss is minimized.
18. The computer program product of claim 16 , wherein training the third end-to-end neural speech recognition model includes:
masking less probable nodes of the output probability lattice of the second end-to-end neural speech recognition model with respect to a minimum probability threshold; and
jointly minimizing (i) the end-to-end neural speech recognition model loss and (ii) a relative divergence over unmasked portions of the output probability lattice of the second end-to-end neural speech recognition model to an output probability lattice of the third end-to-end speech recognition model.
19. A computer processing system for model training, comprising:
a memory device for storing program code; and
a hardware processor device operatively coupled to the memory device for running the program code, the hardware processor device configured for:
training a second end-to-end neural speech recognition model that has a bidirectional encoder to output same symbols from an output probability lattice of the second end-to-end neural speech recognition model as from an output probability lattice of a trained first end-to-end neural speech recognition model having a unidirectional encoder that streams the output data from a recurrent neural network (RNN) transducer; and
building a third end-to-end neural speech recognition model configured to stream output data from an RNN transducer and comprising a unidirectional encoder by training the third end-to-end speech recognition model as a student using the trained second end-to-end neural speech recognition model as a teacher in a knowledge distillation method, with output data being provided offline by the second end-to-end neural network speech recognition model.