Language-agnostic multilingual modeling using effective script normalization
A method includes obtaining a plurality of training data sets each associated with a respective native language and includes a plurality of respective training data samples. For each respective training data sample of each training data set in the respective native language, the method includes transliterating the corresponding transcription in the respective native script into corresponding transliterated text representing the respective native language of the corresponding audio in a target script and associating the corresponding transliterated text in the target script with the corresponding audio in the respective native language to generate a respective normalized training data sample. The method also includes training, using the normalized training data samples, a multilingual end-to-end speech recognition model to predict speech recognition results in the target script for corresponding speech utterances spoken in any of the different native languages associated with the plurality of training data sets.
1. A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:
obtaining a plurality of training data sets each associated with a single respective native language that is different than the single respective native language of the other training data sets, each training data set comprising a plurality of respective training data samples, each training data sample comprising audio spoken in the respective native language and a corresponding transcription of the audio in a respective native script representing the respective native language;
for each respective training data sample of each training data set in the respective native language:
transliterating the corresponding transcription in the respective native script into corresponding transliterated text representing the respective native language of the corresponding audio in a target script, the target script different from the respective native script; and
associating the corresponding transliterated text in the target script with the corresponding audio in the respective native language to generate a respective normalized training data sample, the respective normalized training data sample comprising the audio spoken in the respective native language and the corresponding transliterated text in the target script; and
training, using the normalized training data samples generated from each respective training data sample of each training data set and without providing any language information, a multilingual end-to-end (E2E) automatic speech recognition (ASR) model to predict speech recognition results in the target script for corresponding speech utterances spoken in any of the different native languages associated with the plurality of training data sets.
2. The computer-implemented method of claim 1 , wherein transliterating the corresponding transcription in the respective native script comprises using a finite state transducer (FST) network to transliterate the corresponding transcription in the respective native script into the corresponding transliterated text.
3. The computer-implemented method of claim 1 , wherein transliterating the corresponding transcription in the respective native script into the corresponding transliterated text comprises using a respective transliteration transducer associated with the respective native script to transliterate the corresponding transcription in the respective native script into the corresponding transliterated text in the target script.
4. The computer-implemented method of claim 3 , wherein the respective transliteration transducer associated with the respective native script comprises:
an input transducer configured to map input Unicode symbols in the respective native script to symbols in a pair language model;
a bigram pair language model transducer configured to map between symbols in the respective native script and the target script; and
an output transducer configured to map the symbols in the pair language model to output symbols in the target script.
5. The computer-implemented method of claim 3 , wherein the operations further comprise, prior to transliterating the corresponding transcription in the respective native language, training, using agreement-based data pre-processing, each respective transliteration transducer to only process transliteration pairs that have at least one spelling in the target script of the transliterated text for a given native word that is common across each of the respective native languages associated with the training data sets.
6. The computer-implemented method of claim 3 , wherein the operations further comprise, prior to transliterating the corresponding transcription in the respective native language, training, using frequency-based data pre-processing, each respective transliteration transducer to only process transliteration pairs that have spellings in the target script of the transliterated text for a given native word that satisfy a frequency threshold.
7. The computer-implemented method of claim 1 , wherein transliterating the corresponding transcription in the respective native script into the corresponding transliterated text comprises using a language-independent transliteration transducer to transliterate the corresponding transcription in the respective native script into the corresponding transliterated text in the target script.
8. The computer-implemented method of claim 1 , wherein the multilingual E2E ASR model comprises a sequence-to-sequence neural network.
9. The computer-implemented method of claim 1 , wherein the multilingual E2E ASR model comprises a recurrent neural network transducer (RNN-T).
10. The computer-implemented method of claim 1 , wherein training the multilingual E2E ASR model comprises using a stochastic optimization algorithm to train the multilingual E2E ASR model.
11. The computer-implemented method of claim 1 , wherein the operations further comprise, prior to training the multilingual E2E ASR model, shuffling the normalized training data samples generated from each respective training data sample of each training data set.
12. The computer-implemented method of claim 1 , wherein the operations further comprise, after training the multilingual E2E ASR model, pushing the trained multilingual E2E ASR model to a plurality of user devices, each user device configured to:
capture, using at least one microphone in communication with the user device, an utterance spoken by a respective user of the user device in any combination of the respective native languages associated with the training data sets; and
generate, using the trained multilingual E2E ASR model, a corresponding speech recognition result in the target script for the captured utterance spoken by the respective user.
13. The computer-implemented method of claim 12 , wherein at least one of the plurality of user devices is further configured to transliterate the corresponding speech recognition result in the target script into a transliterated script.
14. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
obtaining a plurality of training data sets each associated with a single respective native language that is different than the single respective native language of the other training data sets, each training data set comprising a plurality of respective training data samples, each training data sample comprising audio spoken in the respective native language and a corresponding transcription of the audio in a respective native script representing the respective native language;
for each respective training data sample of each training data set in the respective native language:
transliterating the corresponding transcription in the respective native script into corresponding transliterated text representing the respective native language of the corresponding audio in a target script, the target script different from the respective native script; and
associating the corresponding transliterated text in the target script with the corresponding audio in the respective native language to generate a respective normalized training data sample, the respective normalized training data sample comprising the audio spoken in the respective native language and the corresponding transliterated text in the target script; and
training, using the normalized training data samples generated from each respective training data sample of each training data set and without providing any language information, a multilingual end-to-end (E2E) automatic speech recognition (ASR) model to predict speech recognition results in the target script for corresponding speech utterances spoken in any of the different native languages associated with the plurality of training data sets.
15. The system of claim 14 , wherein transliterating the corresponding transcription in the respective native script comprises using a finite state transducer (FST) network to transliterate the corresponding transcription in the respective native script into the corresponding transliterated text.
16. The system of claim 14 , wherein transliterating the corresponding transcription in the respective native script into the corresponding transliterated text comprises using a respective transliteration transducer associated with the respective native script to transliterate the corresponding transcription in the respective native script into the corresponding transliterated text in the target script.
17. The system of claim 16 , wherein the respective transliteration transducer associated with the respective native script comprises:
an input transducer configured to map input Unicode symbols in the respective native script to symbols in a pair language model;
a bigram pair language model transducer configured to map between symbols in the respective native script and the target script; and
an output transducer configured to map the symbols in the pair language model to output symbols in the target script.
18. The system of claim 16 , wherein the operations further comprise, prior to transliterating the corresponding transcription in the respective native language, training, using agreement-based data pre-processing, each respective transliteration transducer to only process transliteration pairs that have at least one spelling in the target script of the transliterated text for a given native word that is common across each of the respective native languages associated with the training data sets.
19. The system of claim 16 , wherein the operations further comprise, prior to transliterating the corresponding transcription in the respective native language, training, using frequency-based data pre-processing, each respective transliteration transducer to only process transliteration pairs that have spellings in the target script of the transliterated text for a given native word that satisfy a frequency threshold.
20. The system of claim 14 , wherein transliterating the corresponding transcription in the respective native script into the corresponding transliterated text comprises using a language-independent transliteration transducer to transliterate the corresponding transcription in the respective native script into the corresponding transliterated text in the target script.
21. The system of claim 14 , wherein the multilingual E2E ASR model comprises a sequence-to-sequence neural network.
22. The system of claim 14 , wherein the multilingual E2E ASR model comprises a recurrent neural network transducer (RNN-T).
23. The system of claim 14 , wherein training the multilingual E2E ASR model comprises using a stochastic optimization algorithm to train the multilingual E2E ASR model.
24. The system of claim 14 , wherein the operations further comprise, prior to training the multilingual E2E ASR model, shuffling the normalized training data samples generated from each respective training data sample of each training data set.
25. The system of claim 14 , wherein the operations further comprise, after training the multilingual E2E ASR model, pushing the trained multilingual E2E ASR model to a plurality of user devices, each user device configured to:
capture, using at least one microphone in communication with the user device, an utterance spoken by a respective user of the user device in any combination of the respective native languages associated with the training data sets; and
generate, using the trained multilingual E2E ASR model, a corresponding speech recognition result in the target script for the captured utterance spoken by the respective user.
26. The system of claim 25 , wherein at least one of the plurality of user devices is further configured to transliterate the corresponding speech recognition result in the target script into a transliterated script.