Self-adaptive distillation
A method for distilling one or more trained teacher automatic speech recognition (ASR) models into a multilingual student model includes receiving a plurality of teacher training examples and a plurality of student training examples. The method also includes training one or more teacher automatic speech recognition (ASR) models using the plurality of teacher training examples. Each teacher ASR model is configured to output a respective textual representation of a respective audio input. The method further includes generating a multi-lingual student ASR model by training the multi-lingual student ASR model using the plurality of student training examples and distilling the trained one or more teacher ASR models into the multilingual student ASR model using a tunable distillation loss weight. Each student ASR model is configured to receive an audio input and output a corresponding textual representation of the received audio input.
1. A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising:
receiving a plurality of teacher training examples and a plurality of student training examples;
training one or more teacher automatic speech recognition (ASR) models using the plurality of teacher training examples, each teacher ASR model comprising a recurrent neural network-transducer (RNN-T) architecture and configured to output a respective textual representation of a respective audio input; and
generating a multi-lingual student ASR model comprising an RNN-T architecture by:
training the multi-lingual student ASR model using the plurality of student training examples, the student ASR model configured to receive an audio input and to output a corresponding textual representation of the received audio input; and
distilling the trained one or more teacher ASR models into the multi-lingual student ASR model by applying a tunable weight to a distillation loss of the multi-lingual student ASR model to generate a tunable distillation loss weight, the tunable distillation loss weight comprising a decreasing function based on a first RNN-T loss corresponding to the one or more teacher ASR models and a second RNN-T loss corresponding to the multi-lingual student ASR model.
2. The method of claim 1 , wherein the one or more teacher ASR models are configured to collectively recognize fewer languages than the multi-lingual student ASR model.
3. The method of claim 1 , wherein:
training the multi-lingual student ASR model occurs across n number of training steps; and
the tunable distillation loss weight comprises a decreasing function decreasing based on the n number of training steps.
4. The method of claim 1 , wherein the decreasing function:
decreases the first RNN-T loss corresponding to the one or more teacher ASR models over an instance of time; and
increases the second RNN-T loss corresponding to the multi-lingual student ASR model over the instance of time.
5. The method of claim 1 , wherein each teacher ASR model of the one or more teacher ASR models corresponds to a mono-lingual teacher ASR model.
6. The method of claim 1 , wherein the one or more teacher ASR models corresponds to a single multi-lingual ASR model.
7. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving a plurality of teacher training examples and a plurality of student training examples;
training one or more teacher automatic speech recognition (ASR) models using the plurality of teacher training examples, each teacher ASR model comprising a recurrent neural network-transducer (RNN-T) architecture and configured to output a respective textual representation of a respective audio input; and
generating a multi-lingual student ASR model comprising an RNN-T architecture by:
training the multi-lingual student ASR model using the plurality of student training examples, the student ASR model configured to receive an audio input and to output a corresponding textual representation of the received audio input; and
distilling the trained one or more teacher ASR models into the multi-lingual student ASR model by applying a tunable weight to a distillation loss of the multi-lingual student ASR model to generate a tunable distillation loss weight, the tunable distillation loss weight comprising a decreasing function based on a first RNN-T loss corresponding to the one or more teacher ASR models and a second RNN-T loss corresponding to the multi-lingual student ASR model.
8. The system of claim 7 , wherein the one or more teacher ASR models are configured to collectively recognize fewer languages than the multi-lingual student ASR model.
9. The system of claim 7 , wherein:
training the multi-lingual student ASR model occurs across n number of training steps; and
the tunable distillation loss weight comprises a decreasing function decreasing based on the n number of training steps.
10. The system of claim 7 , wherein the decreasing function:
decreases the first RNN-T loss corresponding to the one or more teacher ASR models over an instance of time; and
increases the second RNN-T loss corresponding to the multi-lingual student ASR model over the instance of time.
11. The system of claim 7 , wherein each teacher ASR model of the one or more teacher ASR models corresponds to a mono-lingual teacher ASR model.
12. The system of claim 7 , wherein the one or more teacher ASR models corresponds to a single multi-lingual ASR model.