Cross-lingual voice cloning for low-resource languages
Techniques for cross-lingual voice cloning for low-resource languages are provided. In an example method, a computing device receives a first speech sample, the first speech sample characterized by a first voice and spoken in a first language and a text input in a second language. The computing device generates, using a first trained ML model trained to encode the text input to an encoded representation that characterizes the text input spoken in a second voice in the second language, the encoded representation of the text input. The computing device generates, using a second trained ML model trained to generate a spectrogram based on the encoded representation of the text input spoken in the second language, the spectrogram of the text input, the spectrogram characterized by the text input spoken in the first voice in the second language. The computing device generates an audio output based on the spectrogram.
1 . A method, comprising:
receiving a first speech sample, the first speech sample characterized by a first voice and spoken in a first language;
receiving a text input in a second language, wherein the second language is a low-resource language;
generating, by a first trained machine learning (“ML”) model trained using low-resource training data, to encode the text input to an encoded representation that characterizes the text input spoken in a second voice in the second language, a first encoded representation of the text input;
generating, by a second trained ML model trained using high-resource training data, to generate a spectrogram based on the first encoded representation of the text input spoken in the second language, a first spectrogram of the text input, the first spectrogram characterized by the text input spoken in the first voice in the second language, wherein the first trained ML model is different from the second trained ML model; and
generating an audio output based on the spectrogram.
2 . The method of claim 1 , wherein training the first trained ML model to encode the text input to the encoded representation that characterizes the text input spoken in the second voice in the second language comprises:
accessing a third trained ML model, wherein the third trained ML model is an automatic speech recognition encoder trained to generate the encoded representation;
generating, by the third trained ML model, the encoded representation of the text input spoken in the second voice in the second language; and
training the first trained ML model to generate the encoded representation generated by the third trained ML model.
3 . The method of claim 1 , further comprising generating an embedded representation of the first speech sample, wherein the embedded representation is generated by a third trained ML model, wherein the third trained ML model is trained to generate the embedded representation using language-independent training data.
4 . The method of claim 1 , wherein the first trained ML model is a transformer comprising one or more transformer blocks and one or more feed-forward blocks.
5 . The method of claim 1 , wherein the second trained ML model is a transformer comprising one or more transformer blocks and one or more feed-forward blocks.
6 . The method of claim 1 , wherein:
the first spectrogram is a Mel spectrogram; and
generating the audio output based on the first spectrogram comprises:
receiving, by a vocoder, the Mel spectrogram;
generating an audio waveform based on the Mel spectrogram; and
playing back the audio waveform using an audio output device.
7 . The method of claim 6 , wherein the vocoder is a trained generative adversarial neural network.
8 . The method of claim 1 , wherein the text input comprises one or more characters, wherein the one or more characters include at least one of a grapheme or a phoneme.
9 . A system comprising:
one or more processors; and
one or more computer-readable storage media storing instructions which, when executed by the one or more processors, cause the one or more processors to perform operations including:
receiving a first speech sample, the first speech sample characterized by a first voice and spoken in a first language;
receiving a text input in a second language, wherein the second language is a low-resource language;
generating, by a first trained ML model trained using low-resource training data, to encode the text input to an encoded representation that characterizes the text input spoken in a second voice in the second language, a first encoded representation of the text input;
generating, by a second trained ML model trained using high-resource training data, to generate a spectrogram based on the first encoded representation of the text input spoken in the second language, a first spectrogram of the text input, the first spectrogram characterized by the text input spoken in the first voice in the second language, wherein the first trained ML model is different from the second trained ML model; and
generating an audio output based on the spectrogram.
10 . The system of claim 9 , wherein the first trained ML model is an automatic speech recognition encoder.
11 . The system of claim 9 , wherein the first trained ML model is a transformer comprising one or more transformer blocks and one or more feed-forward blocks.
12 . The system of claim 9 , wherein the second trained ML model is a transformer comprising one or more transformer blocks and one or more feed-forward blocks.
13 . The system of claim 9 , wherein:
the first spectrogram is a Mel spectrogram; and
generating the audio output based on the first spectrogram comprises:
receiving, by a vocoder, the Mel spectrogram;
generating an audio waveform based on the Mel spectrogram; and
playing back the audio waveform using an audio output device.
14 . The system of claim 13 , wherein the vocoder is a trained generative adversarial neural network.
15 . The system of claim 9 , wherein the instructions further comprise generating an embedded representation of the first speech sample, wherein the embedded representation is generated by a third trained ML model, wherein the third trained ML model is trained to generate the embedded representation using language-independent training data.
16 . A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations including:
receiving a first speech sample, the first speech sample characterized by a first voice and spoken in a first language;
receiving a text input in a second language, wherein the second language is a low-resource language;
generating, by a first trained machine learning (“ML”) model trained using low-resource training data, to encode the text input to an encoded representation that characterizes the text input spoken in a second voice in the second language, a first encoded representation of the text input;
generating, by a second trained ML model trained using high-resource training data, to generate a spectrogram based on the first encoded representation of the text input spoken in the second language, a first spectrogram of the text input, the first spectrogram characterized by the text input spoken in the first voice in the second language, wherein the first trained ML model is different from the second trained ML model; and
generating an audio output based on the spectrogram.
17 . The non-transitory computer-readable medium of claim 16 , wherein training the first trained ML model to encode the text input to the encoded representation that characterizes the text input spoken in the second voice in the second language comprises:
accessing a third trained ML model, wherein the third trained ML model is an automatic speech recognition encoder trained to generate the encoded representation;
generating, by the third trained ML model, the encoded representation of the text input spoken in the second voice in the second language; and
training the first trained ML model to generate the encoded representation generated by the third trained ML model.
18 . The non-transitory computer-readable medium of claim 16 , further comprising generating an embedded representation of the first speech sample, wherein the embedded representation is generated by a third trained ML model, wherein the third trained ML model is trained to generate the embedded representation using language-independent training data.
19 . The non-transitory computer-readable medium of claim 16 , wherein:
the first trained ML model is a first transformer comprising one or more first transformer blocks and one or more first feed-forward blocks; and
the second trained ML model is a second transformer comprising one or more second transformer blocks and one or more second feed-forward blocks.
20 . The non-transitory computer-readable medium of claim 16 , wherein:
the first spectrogram is a Mel spectrogram; and
generating the audio output based on the first spectrogram comprises:
receiving, by a vocoder, the Mel spectrogram, wherein the vocoder is a trained generative adversarial neural network;
generating an audio waveform based on the Mel spectrogram; and
playing back the audio waveform using an audio output device.