Large-scale multilingual speech recognition with a streaming end-to-end model

1. A method comprising:

receiving, at data processing hardware, audio data for an utterance spoken in a particular native language;

obtaining, by the data processing hardware, a language vector identifying the particular native language;

processing, by the data processing hardware, using a multilingual end-to-end (E2E) speech recognition model that uses a recurrent neural network-transducer (RNN-T) architecture comprising an encoder network, the language vector and acoustic features derived from the audio data to generate a transcription for the utterance, the multilingual E2E speech recognition model comprising a plurality of language-specific adaptor modules that include one or more adaptor modules specific to the particular native language and one or more other adaptor modules specific to at least one other native language different than the particular native language, wherein processing the language vector causes the multilingual E2E speech recognition model to activate the one or more adaptor modules specific to the particular native language so that the multilingual E2E speech recognition model only applies the activated one or more adaptor modules specific to the particular native language without applying the one or more other adaptor modules specific to the at least one other native language when processing the acoustic features derived from the audio data to generate the transcription; and

providing, by the data processing hardware, the transcription for output,

wherein the encoder network of the RNN-T architecture comprises:

a plurality of stacked Long Short-Term Memory (LSTM) layers; and

after each LSTM layer, a respective layer comprising a respective subset of the plurality of language-specific adaptor modules, each language-specific adaptor module in the respective layer specific to a different respective native language, wherein one of the language-specific adaptor modules in the respective layer is specific to the particular native language.

2. The method of claim 1 , wherein the language vector comprises a one-hot vector.

3. The method of claim 1 , further comprising, prior to processing the language vector and the acoustic features, generating, by the data processing hardware, using the audio data, a vector representation of the utterance, the vector representation of the utterance comprising the acoustic features derived from the audio data.

4. The method of claim 3 , further comprising:

concatenating, by the data processing hardware, the language vector and the vector representation of the utterance to generate an input vector,

wherein processing the language vector and the acoustic features comprises processing, using the multilingual E2E speech recognition model, the input vector to generate the transcription for the utterance.

5. The method of claim 1 , wherein obtaining the language vector comprises:

identifying the particular native language for the utterance based on a language preference for a user that spoke the utterance; and

generating the language vector based on the identified particular native language.

6. The method of claim 1 , wherein obtaining the language vector comprises:

executing a language identification system configured to identify the particular native language by processing the audio data; and

generating the language vector based on the identified particular native language.

7. The method of claim 1 , wherein the RNN-T architecture comprises:

the encoder network configured to generate, at each of a plurality of time steps, a higher-order feature representation from an input vector, the input vector comprising a concatenation of the language vector and the acoustic features derived from the audio data;

a prediction network configured to process a sequence of previously output non-blank symbols into a dense representation; and

a joint network configured to predict, at each of the plurality of time steps, a probability distribution over possible output labels based on the higher-order feature representation output by the encoder network and the dense representation output by the prediction network.

8. The method of claim 1 , wherein the generated transcription for the utterance is in a respective native script representing the particular native language.

9. The method of claim 1 , wherein the data processing hardware and the multilingual speech recognition model reside on a user device associated with a user that spoke the utterance.

10. The method of claim 1 , wherein the multilingual E2E speech recognition model is trained by a training process, the training process comprises:

obtaining a plurality of training data sets each associated with a respective native language that is different than the respective native languages of the other training data sets, each training data set comprising a plurality of respective training data samples, each training data sample comprising audio data for an utterance spoken in the respective native language, a language identifier identifying the respective native language, and a corresponding transcription of the utterance in a respective native script representing the respective native language;

during a first stage of the training process, training the multilingual E2E speech recognition model on a union of all of the training data sets using a stochastic optimization algorithm; and

during a second stage of the training process after the first stage is complete:

modifying the multilingual E2E speech recognition model to include the plurality of language-specific adaptor modules; and

for each of the one or more adaptor modules that are specific to the particular native language, learning values for a respective set of weights by training the multilingual E2E speech recognition model only on the training data set that is associated with the respective particular native language.

11. The method of claim 10 , wherein the training process executes on a remote computing device in communication with the data processing hardware, the data processing hardware residing on a user device associated with a user that spoke the utterance and configured to execute the multilingual E2E speech recognition model after the training process is complete.

12. A system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:

receiving audio data for an utterance spoken in a particular native language;

obtaining a language vector identifying the particular native language;

processing, using a multilingual end-to-end (E2E) speech recognition model that uses a recurrent neural network-transducer (RNN-T) architecture comprising an encoder network, the language vector and acoustic features derived from the audio data to generate a transcription for the utterance, the multilingual E2E speech recognition model comprising a plurality of language-specific adaptor modules that include one or more adaptor modules specific to the particular native language and one or more other adaptor modules specific to at least one other native language different than the particular native language, wherein processing the language vector causes the multilingual E2E speech recognition model to activate the one or more adaptor modules specific to the particular native language so that the multilingual E2E speech recognition model only applies the activated one or more adaptor modules specific to the particular native language without applying the one or more other adaptor modules specific to the at least one other native language when processing the acoustic features derived from the audio data to generate the transcription; and

providing the transcription for output,

wherein the encoder network of the RNN-T architecture comprises:

a plurality of stacked Long Short-Term Memory (LSTM) layers; and

after each LSTM layer, a respective layer comprising a respective subset of the plurality of language-specific adaptor modules, each language-specific adaptor module in the respective layer specific to a different respective native language, wherein one of the language-specific adaptor modules in the respective layer is specific to the particular native language.

13. The system of claim 12 , wherein the language vector comprises a one-hot vector.

14. The system of claim 12 , wherein the operations further comprise, prior to processing the language vector and the acoustic features, generating, using the audio data, a vector representation of the utterance, the vector representation of the utterance comprising the acoustic features derived from the audio data.

15. The system of claim 12 , wherein the operations further comprise:

concatenating the language vector and the vector representation of the utterance to generate an input vector,

wherein processing the language vector and the acoustic features comprises processing, using the multilingual E 2 E speech recognition model, the input vector to generate the transcription for the utterance.

16. The system of claim 12 , wherein obtaining the language vector comprises:

identifying the particular native language for the utterance based on a language preference for a user that spoke the utterance; and

generating the language vector based on the identified particular native language.

17. The system of claim 12 , wherein obtaining the language vector comprises:

executing a language identification system configured to identify the particular native language by processing the audio data; and

generating the language vector based on the identified particular native language.

18. The system of claim 12 , wherein the RNN-T architecture comprises:

the encoder network configured to generate, at each of a plurality of time steps, a higher-order feature representation from an input vector, the input vector comprising a concatenation of the language vector and the acoustic features derived from the audio data;

a prediction network configured to process a sequence of previously output non-blank symbols into a dense representation; and

a joint network configured to predict, at each of the plurality of time steps, a probability distribution over possible output labels based on the higher-order feature representation output by the encoder network and the dense representation output by the prediction network.

19. The system of claim 12 , wherein the generated transcription for the utterance is in a respective native script representing the particular native language.

20. The system of claim 12 , wherein the data processing hardware and the multilingual speech recognition model reside on a user device associated with a user that spoke the utterance.

21. The system of claim 10 , wherein the multilingual E2E speech recognition model is trained by a training process, the training process comprises:

obtaining a plurality of training data sets each associated with a respective native language that is different than the respective native languages of the other training data sets, each training data set comprising a plurality of respective training data samples, each training data sample comprising audio data for an utterance spoken in the respective native language, a language identifier identifying the respective native language, and a corresponding transcription of the utterance in a respective native script representing the respective native language;

during a first stage of the training process, training the multilingual E2E speech recognition model on a union of all of the training data sets using a stochastic optimization algorithm; and

during a second stage of the training process after the first stage is complete:

modifying the multilingual E2E speech recognition model to include the plurality of language-specific adaptor modules; and

for each of the one or more adaptor modules that are specific to the particular native language, learning values for a respective set of weights by training the multilingual E2E speech recognition model only on the training data set that is associated with the respective particular native language.

22. The system of claim 21 , wherein the training process executes on a remote computing device in communication with the data processing hardware, the data processing hardware residing on a user device associated with a user that spoke the utterance and configured to execute the multilingual E2E speech recognition model after the training process is complete.