Unsupervised parallel tacotron non-autoregressive and controllable text-to-speech

1. A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations for training a non-autoregressive text-to-speech (TTS) model, the operations comprising:

obtaining a sequence representation of an encoded text sequence concatenated with a variational embedding;

using a duration model network:

predicting, based on the sequence representation, a phoneme duration for each phoneme represented by the encoded text sequence;

based on the predicted phoneme durations:

learning, using a first function conditioned on the sequence representation, an interval representation matrix; and

learning, using a second function conditioned on the sequence representation, an auxiliary attention context representation;

determining a product of the interval representation matrix and the sequence representation;

determining an Einstein summation (einsum) of the interval representation matrix and the auxiliary attention context representation; and

upsampling, based on summing the product of the interval representation matrix and the sequence representation and a projection of the einsum, the sequence representation into an upsampled output specifying a number of frames;

generating, as output from a spectrogram decoder comprising a stack of one or more self-attention blocks, based on the upsampled output, one or more predicted mel-frequency spectrogram sequences for the encoded text sequence;

determining a final spectrogram loss based on the one or more predicted mel-frequency spectrogram sequences and a reference mel-frequency spectrogram sequence; and

training the TTS model based on the final spectrogram loss.

2. The computer-implemented method of claim 1 , wherein the first function and the second function each comprise a respective multi-layer perception-based learnable function.

3. The computer-implemented method of claim 1 , wherein the operations further comprise:

determining a global phoneme duration loss based on the predicted phoneme durations and an average phoneme duration,

wherein training the TTS model is further based on the global phoneme duration loss.

4. The computer implemented method of claim 3 , wherein training the TTS model based on the final spectrogram loss and the global phoneme duration loss comprises training the duration model network to predict the phoneme duration for each phoneme without using supervised phoneme duration labels extracted from an external aligner.

5. The computer-implemented method of claim 1 , wherein the operations further comprise, using the duration model network:

based on the predicted phoneme durations, generating, for each phoneme represented by the encoded text sequence, respective start and end boundaries;

mapping, based on a number of phonemes represented by the encoded text sequence and a number of reference frames in the reference mel-frequency spectrogram sequence, the respective start and end boundaries generated for each phoneme into respective grid matrices,

wherein learning the interval representation is based on the respective grid matrices mapped from the start and end boundaries; and

wherein learning the auxiliary attention context representation is based on the respective grid matrices mapped from the start and end boundaries.

6. The computer-implemented method of claim 1 , wherein the operations further comprise:

receiving training data including a reference audio signal and a corresponding input text sequence, the reference audio signal comprising a spoken utterance and the input text sequence corresponds to a transcript of the reference audio signal;

encoding, using a residual encoder, the reference audio signal into a variational embedding, the variational embedding disentangling style/prosody information from the reference audio signal; and

encoding, using a text encoder, the input text sequence into the encoded text sequence.

7. The computer-implemented method of claim 6 , wherein:

the residual encoder comprises a global variational autoencoder (VAE); and

encoding the reference audio signal into the variational embedding comprises:

sampling the reference mel-frequency spectrogram sequence from the reference audio signal; and

encoding, using the global VAE, the reference mel-frequency spectrogram sequence into the variational embedding.

8. The computer-implemented method of claim 6 , wherein:

the residual encoder comprises a phoneme-level fine-grained variational autoencoder (VAE); and

encoding the reference audio signal into the variational embedding comprises:

sampling the reference mel-frequency spectrogram sequence from the reference audio signal;

aligning the reference mel-frequency spectrogram sequence with each phoneme in a sequence of phonemes extracted from the input text sequence; and

encoding, using the phoneme-level fine-grained VAE, based on aligning the reference mel-frequency spectrogram sequence with each phoneme in the sequence of phonemes, a sequence of phoneme-level variational embeddings.

9. The computer-implemented method of claim 6 , wherein the residual encoder comprises a stack of lightweight convolution (LConv) blocks, each LConv block in the stack of LConv blocks comprises:

a gated linear unit (GLU) layer;

a LConv layer configured to receive an output of the GLU layer;

a residual connection configured to concatenate an output of the LConv layer with an input to the GLU layer; and

a final feedforward layer configured to receive, as input, the residual connection concatenating the output of the LConv layer with the input to the GLU layer.

10. The computer-implemented method of claim 6 , wherein the operations further comprise:

concatenating the encoded text sequence, the variational embedding, and a reference speaker embedding representing an identity of a reference speaker that uttered the reference audio signal; and

generating the sequence representation based on the duration modeling network receiving, as input, the concatenation of the encoded text sequence, the variational embedding, and the reference speaker embedding.

11. The computer-implemented method of claim 6 , wherein:

the input text sequence includes a sequence of phonemes; and

encoding the input text sequence into the encoded text sequence comprises:

receiving, from a phoneme look-up table, a respective embedding of each phoneme in the sequence of phonemes;

for each phoneme in the sequence of phonemes, processing, using an encoder pre-net neural network of the text encoder, the respective embedding to generate a respective transformed embedding of the phoneme;

processing, using a bank of convolutional blocks, the respective transformed embeddings to generate convolution outputs; and

processing, using a stack of self-attention blocks, the convolution outputs to generate the encoded text sequence.

12. The computer-implemented method of claim 1 , wherein each self-attention block in the stack of self-attention blocks comprises an identical lightweight convolution (LConv) block.

13. The computer-implemented method of claim 1 , wherein each self-attention block in the stack of self-attention blocks comprises an identical transformer block.

14. A system for training a non-autoregressive text-to-speech (TTS) model, the system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed by the data processing hardware cause the data processing hardware to perform operations comprising:

obtaining a sequence representation of an encoded text sequence concatenated with a variational embedding;

using a duration model network:

predicting, based on the sequence representation, a phoneme duration for each phoneme represented by the encoded text sequence;

based on the predicted phoneme durations:

learning, using a first function conditioned on the sequence representation, an interval representation matrix; and

learning, using a second function conditioned on the sequence representation, an auxiliary attention context representation;

determining a product of the interval representation matrix and the sequence representation;

determining an Einstein summation (einsum) of the interval representation matrix and the auxiliary attention context representation; and

upsampling, based on summing the product of the interval representation matrix and the sequence representation and a projection of the einsum, the sequence representation into an upsampled output specifying a number of frames;

generating, as output from a spectrogram decoder comprising a stack of one or more self-attention blocks, based on the upsampled output, one or more predicted mel-frequency spectrogram sequences for the encoded text sequence;

determining a final spectrogram loss based on the one or more predicted mel-frequency spectrogram sequences and a reference mel-frequency spectrogram sequence; and

training the TTS model based on the final spectrogram loss.

15. The system of claim 14 , wherein the first function and the second function each comprise a respective multi-layer perception-based learnable function.

16. The system of claim 14 , wherein the operations further comprise:

determining a global phoneme duration loss based on the predicted phoneme durations and an average phoneme duration,

wherein training the TTS model is further based on the global phoneme duration loss.

17. The system of claim 16 , wherein training the TTS model based on the final spectrogram loss and the global phoneme duration loss comprises training the duration model network to predict the phoneme duration for each phoneme without using supervised phoneme duration labels extracted from an external aligner.

18. The system of claim 14 , wherein the operations further comprise, using the duration model network:

based on the predicted phoneme durations, generating, for each phoneme represented by the encoded text sequence, respective start and end boundaries;

mapping, based on a number of phonemes represented by the encoded text sequence and a number of reference frames in the reference mel-frequency spectrogram sequence, the respective start and end boundaries generated for each phoneme into respective grid matrices,

wherein learning the interval representation is based on the respective grid matrices mapped from the start and end boundaries; and

wherein learning the auxiliary attention context representation is based on the respective grid matrices mapped from the start and end boundaries.

19. The system of claim 14 , wherein the operations further comprise:

receiving training data including a reference audio signal and a corresponding input text sequence, the reference audio signal comprising a spoken utterance and the input text sequence corresponds to a transcript of the reference audio signal;

encoding, using a residual encoder, the reference audio signal into a variational embedding, the variational embedding disentangling style/prosody information from the reference audio signal; and

encoding, using a text encoder, the input text sequence into the encoded text sequence.

20. The system of claim 19 , wherein:

the residual encoder comprises a global variational autoencoder (VAE); and

encoding the reference audio signal into the variational embedding comprises:

sampling the reference mel-frequency spectrogram sequence from the reference audio signal; and

encoding, using the global VAE, the reference mel-frequency spectrogram sequence into the variational embedding.

21. The system of claim 19 , wherein:

the residual encoder comprises a phoneme-level fine-grained variational autoencoder (VAE); and

encoding the reference audio signal into the variational embedding comprises:

sampling the reference mel-frequency spectrogram sequence from the reference audio signal;

aligning the reference mel-frequency spectrogram sequence with each phoneme in a sequence of phonemes extracted from the input text sequence; and

encoding, using the phoneme-level fine-grained VAE, based on aligning the reference mel-frequency spectrogram sequence with each phoneme in the sequence of phonemes, a sequence of phoneme-level variational embeddings.

22. The system of claim 19 , wherein the residual encoder comprises a stack of lightweight convolution (LConv) blocks, each LConv block in the stack of LConv blocks comprises:

a gated linear unit (GLU) layer;

a LConv layer configured to receive an output of the GLU layer;

a residual connection configured to concatenate an output of the LConv layer with an input to the GLU layer; and

a final feedforward layer configured to receive, as input, the residual connection concatenating the output of the LConv layer with the input to the GLU layer.

23. The system of claim 19 , wherein the operations further comprise:

concatenating the encoded text sequence, the variational embedding, and a reference speaker embedding representing an identity of a reference speaker that uttered the reference audio signal; and

generating sequence representation based on the duration modeling network receiving, as input, the concatenation of the encoded text sequence, the variational embedding, and the reference speaker embedding.

24. The system of claim 19 , wherein:

the input text sequence includes a sequence of phonemes; and

encoding the input text sequence into the encoded text sequence comprises:

receiving, from a phoneme look-up table, a respective embedding of each phoneme in the sequence of phonemes;

for each phoneme in the sequence of phonemes, processing, using an encoder pre-net neural network of the text encoder, the respective embedding to generate a respective transformed embedding of the phoneme;

processing, using a bank of convolutional blocks, the respective transformed embeddings to generate convolution outputs; and

processing, using a stack of self-attention blocks, the convolution outputs to generate the encoded text sequence.

25. The system of claim 14 , wherein each self-attention block in the stack of self-attention blocks comprises an identical lightweight convolution (LConv) block.

26. The system of claim 14 , wherein each self-attention block in the stack of self-attention blocks comprises an identical transformer block.