Speaker adaptation for attention-based encoder-decoder

1. A system comprising:

a processing unit; and

a memory storage device including program code that when executed by the processing unit causes to the system to:

train a speaker-independent attention-based encoder-decoder model to classify output units based on first input speech frames, the trained speaker-independent attention-based encoder-decoder model associated with a first output distribution;

adapt the speaker-independent attention-based encoder-decoder model by training the speaker-independent attention-based encoder-decoder model to classify output tokens based on input speech frames of a target speaker, and simultaneously training the speaker-independent attention-based encoder-decoder model to maintain a similarity between the first output distribution and a second output distribution of the adapted speaker-independent attention-based encoder-decoder model; and

perform automatic speech recognition using the adapted speaker-independent attention-based encoder-decoder model.

2. The system of claim 1 , wherein adaptation of the speaker-independent attention-based encoder-decoder model comprises:

initializing of a speaker-dependent attention-based encoder-decoder model with an encoder, an attention network and a decoder of the trained speaker-independent attention-based encoder-decoder model;

training of the speaker-dependent attention-based encoder-decoder model to minimize an output token classification loss with respect to the input speech frames of the target speaker, and simultaneously training the speaker-dependent attention-based encoder-decoder model to minimize a divergence measure between the first output distribution and the second output.

3. The system of claim 2 , wherein the divergence measure is a Kullback-Leibler divergence.

4. The system of claim 1 , wherein adaptation of the speaker-independent attention-based encoder-decoder model comprises:

initializing of a speaker-dependent attention-based encoder-decoder model with an encoder, an attention network, lower decoder layers and upper decoder layers of the trained speaker-independent attention-based encoder-decoder model;

initializing of a discriminator to receive data from the lower decoder layers of the speaker-dependent attention-based encoder-decoder model and data from the lower decoder layers of the speaker-independent attention-based encoder-decoder model and to output a prediction of whether received data was generated by the speaker-dependent attention-based encoder-decoder model; and

simultaneous training, based on the input speech frames of the target speaker, of the speaker-dependent attention-based encoder-decoder model to minimize an output token classification loss, of the discriminator to minimize a discrimination loss, and of the encoder, the attention network, and the lower decoder layers of the speaker-dependent attention-based encoder-decoder model to maximize the discrimination loss.

5. The system of claim 1 , wherein the output tokens are word/subword units, and wherein adaptation of the speaker-independent attention-based encoder-decoder model comprises:

initializing of an auxiliary attention-based encoder-decoder model with an encoder of the trained speaker-independent attention-based encoder-decoder model, a character-level attention network, and a character-level decoder;

training of the character-level attention network and the character-level decoder based on the input speech frames of the target speaker to minimize a character-level classification loss; and

training of the encoder of the trained speaker-independent attention-based encoder-decoder model based on the input speech frames of the target speaker to jointly minimize the word/subword unit classification loss of the trained speaker-independent attention-based encoder-decoder model and the character-level classification loss of the auxiliary attention-based encoder-decoder model.

6. The system of claim 5 , wherein parameters of a word/subword unit attention network of the trained speaker-independent attention-based encoder-decoder model, parameters of a word/subword unit decoder network of the trained speaker-independent attention-based encoder-decoder model, parameters of the trained character-level attention network, and parameters of the trained character-level decoder are fixed during training of the encoder of the trained speaker-independent attention-based encoder-decoder model based on the input speech frames of the target speaker to jointly minimize the word/subword unit classification loss of the trained speaker-independent attention-based encoder-decoder model and the character-level classification loss of the auxiliary attention-based encoder-decoder model.

7. The system of claim 6 , wherein automatic speech recognition is performed using the trained encoder, the word/subword unit attention network of the trained speaker-independent attention-based encoder-decoder model, and the word/subword unit decoder of the trained speaker-independent attention-based encoder-decoder model.

8. A computer-implemented method comprising:

training a first attention-based encoder-decoder model for automatic speech recognition, the trained speaker-independent attention-based encoder-decoder model associated with a first output distribution;

adapting the first attention-based encoder-decoder model for automatic speech recognition of speech of a target speaker, and to maintain a similarity between the first output distribution and a second output distribution of the adapted attention-based encoder-decoder model; and

performing automatic speech recognition using the adapted speaker-independent attention-based encoder-decoder model.

9. The method of claim 8 , wherein adapting the speaker-independent attention-based encoder-decoder model comprises:

initializing a speaker-dependent attention-based encoder-decoder model with an encoder, an attention network and a decoder of the trained first attention-based encoder-decoder model;

training the speaker-dependent attention-based encoder-decoder model to minimize an output token classification loss with respect to input speech frames of the target speaker, and simultaneously training the speaker-dependent attention-based encoder-decoder model to minimize a divergence measure between the first output distribution and the second output distribution.

10. The method of claim 9 , wherein the divergence measure is a Kullback-Leibler divergence.

11. The method of claim 8 , wherein adapting the first speaker-independent attention-based encoder-decoder model comprises:

initializing a speaker-dependent attention-based encoder-decoder model with an encoder, an attention network, lower decoder layers and upper decoder layers of the trained first attention-based encoder-decoder model;

initializing a discriminator to receive data from the lower decoder layers of the speaker-dependent attention-based encoder-decoder model and data from the lower decoder layers of the first attention-based encoder-decoder model and to output a prediction of whether received data was generated by the speaker-dependent attention-based encoder-decoder model; and

simultaneous training, based on the input speech frames of the target speaker, of the speaker-dependent attention-based encoder-decoder model to minimize an output token classification loss, of the discriminator to minimize a discrimination loss, and of the encoder, the attention network, and the lower decoder layers of the speaker-dependent attention-based encoder-decoder model to maximize the discrimination loss.

12. The method of claim 8 , wherein adaptation of the first attention-based encoder-decoder model comprises:

initializing of an auxiliary attention-based encoder-decoder model with an encoder of the trained first attention-based encoder-decoder model, a character-level attention network, and a character-level decoder;

training of the character-level attention network and the character-level decoder based on input speech frames of the target speaker to minimize a character-level classification loss; and

training of the encoder of the trained first attention-based encoder-decoder model based on the input speech frames of the target speaker to jointly minimize a word/subword unit classification loss of the trained first attention-based encoder-decoder model and the character-level classification loss of the auxiliary attention-based encoder-decoder model.

13. The method of claim 12 , wherein parameters of a word/subword unit attention network of the trained speaker-independent attention-based encoder-decoder model, parameters of a word/subword unit decoder network of the trained speaker-independent attention-based encoder-decoder model, parameters of the trained character-level attention network, and parameters of the trained character-level decoder are fixed during training of the encoder of the trained speaker-independent attention-based encoder-decoder model based on the input speech frames of the target speaker to jointly minimize the word/subword unit classification loss of the trained speaker-independent attention-based encoder-decoder model and the character-level classification loss of the auxiliary attention-based encoder-decoder model.

14. The method of claim 13 , wherein automatic speech recognition is performed using the trained encoder, the word/subword unit attention network of the trained speaker-independent attention-based encoder-decoder model, and the word/subword unit decoder of the trained speaker-independent attention-based encoder-decoder model.

15. A non-transient, computer-readable medium storing program code to be executed by a processing unit to provide:

a first attention-based encoder-decoder model to classify output tokens based on input speech frames, the first attention-based encoder-decoder model associated with a first output distribution; and

a second attention-based encoder-decoder model to classify output tokens based on input speech frames, the second attention-based encoder-decoder model being associated with a second output distribution,

wherein the second attention-based encoder-decoder model is trained to classify output tokens based on input speech frames of a target speaker, and simultaneously trained to maintain a similarity between the first output distribution and a second output distribution of the second attention-based encoder-decoder model.

16. The medium of claim 15 , wherein the second attention-based encoder-decoder model is initialized with an encoder, an attention network and a decoder of the first attention-based encoder-decoder model, and

wherein training of the second attention-based encoder-decoder mode comprises training of the second attention-based encoder-decoder model to minimize an output token classification loss with respect to the input speech frames of the target speaker, and simultaneously training the second attention-based encoder-decoder model to minimize a divergence measure between the first output distribution and the second output distribution.

17. The medium of claim 16 , wherein the divergence measure is a Kullback-Leibler divergence.

18. The medium of claim 15 , wherein the second attention-based encoder-decoder model is initialized with an encoder, an attention network, lower decoder layers and upper decoder layers of the first attention-based encoder-decoder model;

wherein training of the second attention-based encoder-decoder model comprises:

initializing a discriminator to receive data from the lower decoder layers of the second attention-based encoder-decoder model and data from the lower decoder layers of the first attention-based encoder-decoder model and to output a prediction of whether received data was generated by the second attention-based encoder-decoder model; and

simultaneous training, based on the input speech frames of the target speaker, of the second attention-based encoder-decoder model to minimize an output token classification loss, of the discriminator to minimize a discrimination loss, and of the encoder, the attention network, and the lower decoder layers of the second attention-based encoder-decoder model to maximize the discrimination loss.

19. The medium of claim 15 , wherein the output tokens are word/subword units, the second attention-based encoder-decoder model is initialized with an encoder, an attention network, and a decoder of the first attention-based encoder-decoder model, and

wherein training of the second attention-based encoder-decoder model comprises:

initializing an auxiliary attention-based encoder-decoder model with an encoder of the first attention-based encoder-decoder model, a character-level attention network, and a character-level decoder;

training the character-level attention network and the character-level decoder based on the input speech frames of the target speaker to minimize a character-level classification loss; and

training of the encoder of the second attention-based encoder-decoder model based on the input speech frames of the target speaker to jointly minimize a word/subword unit classification loss of the second attention-based encoder-decoder model and the character-level classification loss of the auxiliary attention-based encoder-decoder model.

20. The medium of claim 19 , wherein parameters of a word/subword unit attention network of the second attention-based encoder-decoder model, parameters of a word/subword unit decoder network of the second attention-based encoder-decoder model, parameters of the trained character-level attention network, and parameters of the trained character-level decoder are fixed during training of the encoder of the second attention-based encoder-decoder model based on the input speech frames of the target speaker to jointly minimize the word/subword unit classification loss of the second attention-based encoder-decoder model and the character-level classification loss of the auxiliary attention-based encoder-decoder model.