Parameter-efficient model reprogramming for cross-lingual speech recognition

1 . A computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations comprising:

obtaining an automatic speech recognition model (ASR) model, the ASR model trained on an initial training dataset to recognize speech in a first language;

receiving an adaptation training dataset comprising one or more transcribed training utterances in a second language different than the first language, each respective transcribed training utterance in the adaptation training dataset paired with a corresponding transcription;

integrating an encoder of the ASR model with an input reprogramming module and a latent reprogramming module, wherein the input reprogramming module is configured to augment each training utterance in the adaptation training dataset with a first trainable noise, and the latent reprogramming module is configured to augment a projection output with a second trainable noise; and

adapting the ASR model integrated with the input reprogramming module and the latent reprogramming module to learn how to recognize speech in the second language by training the input reprogramming module and the latent reprogramming module on the adaptation training dataset while parameters of the ASR model are frozen.

2 . The computer-implemented method of claim 1 , wherein training the input reprogramming module and the latent reprogramming module on the adaptation training dataset comprises, for each respective training utterance:

augmenting, using the input reprogramming module, the respective transcribed training utterance with the first trainable noise;

augmenting, using the latent reprogramming module, the projection output with the second trainable noise, the projection output corresponding to the respective augmented transcribed training utterance;

generating, using the ASR model integrated with the input reprogramming module and the latent reprogramming module, a corresponding speech recognition result for the respective transcribed training utterance based on the augmented projection output;

determining a supervised loss based on the corresponding speech recognition result for the respective transcribed training utterance and the corresponding transcription; and

training the input reprogramming module and the latent reprogramming module based on the supervised loss.

3 . The computer-implemented method of claim 2 , wherein the adaptation training dataset further comprises additional transcribed training utterances in a plurality of other languages, each other languages of the plurality of other languages different than the first language.

4 . The computer-implemented method of claim 2 , wherein augmenting the respective transcribed training utterance with the first trainable noise comprises:

extracting acoustic feature-dependent noise from the respective transcribed training utterance;

generating acoustic feature-independent noise; and

determining the first trainable noise based on the acoustic feature-dependent noise and the acoustic feature-independent noise.

5 . The computer-implemented method of claim 2 , wherein augmenting the respective transcribed training utterance with the first trainable noise minimizes a distance between a distribution of the initial training dataset and a distribution of the adaptation training dataset.

6 . The computer-implemented method of claim 2 , wherein augmenting the projection output with the second trainable noise comprises:

extracting latent feature-dependent noise from the projection output;

generating latent feature-independent noise; and

determining the second trainable noise using the latent feature-dependent noise and the latent feature-independent noise.

7 . The computer-implemented method of claim 2 , wherein the ASR model integrated with the input reprogramming module and the latent reprogramming module comprises:

an encoder comprising:

an input layer configured to:

receive the respective transcribed augmented training utterance from the input reprogramming module; and

generate a corresponding projection output; and

one or more encoder blocks configured to:

receive the corresponding projection output generated by the input layer; and

generate a corresponding higher order feature representation;

a prediction network configured to:

receive a sequence of non-blank symbols output by a final Softmax layer; and

generate a dense representation; and

a joint network configured to:

receive the higher order feature representation generated by the encoder and the dense representation generated by the prediction network; and

generate the corresponding speech recognition result for the respective augmented transcribed training utterance using a set of pre-trained multilingual graphemes.

8 . The computer-implemented method of claim 1 , wherein:

the ASR model comprises an encoder comprising one or more encoder blocks each including a respective encoder layer; and

the latent reprogramming module comprises a stack of reprogramming layers, each respective reprogramming layer in the stack of reprogramming layers comprising a corresponding bridged connection with a respective subsequent reprogramming layer in the stack of reprogramming layers.

9 . The computer-implemented method of claim 8 , wherein integrating the ASR model with the input reprogramming module and the latent reprogramming module comprises inserting the stack of reprogramming layers into the encoder, each respective reprogramming layer in the stack of reprogramming layers disposed between each encoder block of the one or more encoder blocks.

10 . The computer-implemented method of claim 1 , wherein the adaptation training dataset further comprises one or more untranscribed training utterances in the second language, each respective untranscribed utterance in the adaptation training dataset not paired with any corresponding transcription.

11 . The computer-implemented method of claim 10 , wherein the operations further comprise:

determining an unsupervised loss using the one or more untranscribed training utterances; and

training the input reprogramming module and the latent reprogramming module jointly on the unsupervised loss and the supervised loss.

12 . A system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:

obtaining an automatic speech recognition model (ASR) model, the ASR model trained on an initial training dataset to recognize speech in a first language;

receiving an adaptation training dataset comprising one or more transcribed training utterances in a second language different than the first language, each respective transcribed training utterance in the adaptation training dataset paired with a corresponding transcription;

integrating an encoder of the ASR model with an input reprogramming module and a latent reprogramming module, wherein the input reprogramming module is configured to augment each training utterance in the adaptation training dataset with a first trainable noise, and the latent reprogramming module is configured to augment a projection output with a second trainable noise; and

adapting the ASR model integrated with the input reprogramming module and the latent reprogramming module to learn how to recognize speech in the second language by training the input reprogramming module and the latent reprogramming module on the adaptation training dataset while parameters of the ASR model are frozen.

13 . The system of claim 12 , wherein training the input reprogramming module and the latent reprogramming module on the adaptation training dataset comprises, for each respective training utterance:

augmenting, using the input reprogramming module, the respective transcribed training utterance with the first trainable noise;

augmenting, using the latent reprogramming module, the projection output with the second trainable noise, the projection output corresponding to the respective augmented transcribed training utterance;

generating, using the ASR model integrated with the input reprogramming module and the latent reprogramming module, a corresponding speech recognition result for the respective transcribed training utterance based on the augmented projection output;

determining a supervised loss based on the corresponding speech recognition result for the respective transcribed training utterance and the corresponding transcription; and

training the input reprogramming module and the latent reprogramming module based on the supervised loss.

14 . The system of claim 13 , wherein the adaptation training dataset further comprises additional transcribed training utterances in a plurality of other languages, each other languages of the plurality of other languages different than the first language.

15 . The system of claim 13 , wherein augmenting the respective transcribed training utterance with the first trainable noise comprises:

extracting acoustic feature-dependent noise from the respective transcribed training utterance;

generating acoustic feature-independent noise; and

determining the first trainable noise based on the acoustic feature-dependent noise and the acoustic feature-independent noise.

16 . The system of claim 13 , wherein augmenting the respective transcribed training utterance with the first trainable noise minimizes a distance between a distribution of the initial training dataset and a distribution of the adaptation training dataset.

17 . The system of claim 13 , wherein augmenting the projection output with the second trainable noise comprises:

extracting latent feature-dependent noise from the projection output;

generating latent feature-independent noise; and

determining the second trainable noise using the latent feature-dependent noise and the latent feature-independent noise.

18 . The system of claim 13 , wherein the ASR model integrated with the input reprogramming module and the latent reprogramming module comprises:

an encoder comprising:

an input layer configured to:

receive the respective transcribed augmented training utterance from the input reprogramming module; and

generate a corresponding projection output; and

one or more encoder blocks configured to:

receive the corresponding projection output generated by the input layer; and

generate a corresponding higher order feature representation;

a prediction network configured to:

receive a sequence of non-blank symbols output by a final Softmax layer; and

generate a dense representation; and

a joint network configured to:

receive the higher order feature representation generated by the encoder and the dense representation generated by the prediction network; and

generate the corresponding speech recognition result for the respective augmented transcribed training utterance using a set of pre-trained multilingual graphemes.

19 . The system of claim 12 , wherein:

the ASR model comprises an encoder comprising one or more encoder blocks each including a respective encoder layer; and

the latent reprogramming module comprises a stack of reprogramming layers, each respective reprogramming layer in the stack of reprogramming layers comprising a corresponding bridged connection with a respective subsequent reprogramming layer in the stack of reprogramming layers.

20 . The system of claim 19 , wherein integrating the ASR model with the input reprogramming module and the latent reprogramming module comprises inserting the stack of reprogramming layers into the encoder, each respective reprogramming layer in the stack of reprogramming layers disposed between each encoder block of the one or more encoder blocks.

21 . The system of claim 12 , wherein the adaptation training dataset further comprises one or more untranscribed training utterances in the second language, each respective untranscribed utterance in the adaptation training dataset not paired with any corresponding transcription.

22 . The system of claim 21 , wherein the operations further comprise:

determining an unsupervised loss using the one or more untranscribed training utterances; and

training the input reprogramming module and the latent reprogramming module jointly on the unsupervised loss and the supervised loss.