Granted Patent B2

US 12,249,315 · App. 18/499,031 · Granted Mar 11, 2025

Inventors: Isaac Elias (Mountain View, CA); Byungha Chun (Tokyo, JP); Jonathan Shen (Mountain View, CA); Ye Jia (Mountain View, CA); Yu Zhang (Mountain View, CA); Yonghui Wu (Fremont, CA)

Assignee: Google LLC

G10L13/08G10L13/04

Abstract

Claims (96)

1. A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations for training a non-autoregressive text-to-speech (TTS) model, the operations comprising:

obtaining a sequence representation of an encoded text sequence concatenated with a variational embedding;

using a duration model network:

predicting, based on the sequence representation, a phoneme duration for each phoneme represented by the encoded text sequence;

based on the predicted phoneme durations, generating, for each phoneme represented by the encoded text sequence, respective start and end boundaries;

mapping, based on a number of phonemes represented by the encoded text sequence and a number of reference frames in a reference mel-frequency spectrogram sequence, the respective start and end boundaries generated for each phoneme into respective grid matrices;

based on the respective grid matrices mapped from the start and end boundaries, learning, using a first function conditioned on the sequence representation, an interval representation matrix;

determining a product of the interval representation matrix and the sequence representation; and

upsampling, based on the product of the interval representation matrix and the sequence representation, the sequence representation into an upsampled output specifying a number of frames;

generating, as output from a spectrogram decoder comprising a stack of one or more self-attention blocks, based on the upsampled output, one or more predicted mel-frequency spectrogram sequences for the encoded text sequence;

determining a final spectrogram loss based on the one or more predicted mel-frequency spectrogram sequences and the reference mel-frequency spectrogram sequence; and

training the TTS model based on the final spectrogram loss.

2. The computer-implemented method of claim 1 , wherein the first function comprises a respective multi-layer perception-based learnable function.

3. The computer-implemented method of claim 1 , wherein the operations further comprise:

determining a global phoneme duration loss based on the predicted phoneme durations and an average phoneme duration,

wherein training the TTS model is further based on the global phoneme duration loss.

4. The computer implemented method of claim 3 , wherein training the TTS model based on the final spectrogram loss and the global phoneme duration loss comprises training the duration model network to predict the phoneme duration for each phoneme without using supervised phoneme duration labels extracted from an external aligner.

5. The computer-implemented method of claim 1 , wherein the operations further comprise:

receiving training data including a reference audio signal and a corresponding input text sequence, the reference audio signal comprising a spoken utterance and the input text sequence corresponds to a transcript of the reference audio signal;

encoding, using a residual encoder, the reference audio signal into a variational embedding, the variational embedding disentangling style/prosody information from the reference audio signal; and

encoding, using a text encoder, the input text sequence into the encoded text sequence.

6. The computer-implemented method of claim 5 , wherein:

the residual encoder comprises a global variational autoencoder (VAE); and

encoding the reference audio signal into the variational embedding comprises:

sampling the reference mel-frequency spectrogram sequence from the reference audio signal; and

encoding, using the global VAE, the reference mel-frequency spectrogram sequence into the variational embedding.

7. The computer-implemented method of claim 5 , wherein:

the residual encoder comprises a phoneme-level fine-grained variational autoencoder (VAE); and

encoding the reference audio signal into the variational embedding comprises:

sampling the reference mel-frequency spectrogram sequence from the reference audio signal;

aligning the reference mel-frequency spectrogram sequence with each phoneme in a sequence of phonemes extracted from the input text sequence; and

encoding, using the phoneme-level fine-grained VAE, based on aligning the reference mel-frequency spectrogram sequence with each phoneme in the sequence of phonemes, a sequence of phoneme-level variational embeddings.

8. The computer-implemented method of claim 5 , wherein the residual encoder comprises a stack of lightweight convolution (LConv) blocks, each LConv block in the stack of LConv blocks comprises:

a gated linear unit (GLU) layer;

a LConv layer configured to receive an output of the GLU layer;

a residual connection configured to concatenate an output of the LConv layer with an input to the GLU layer; and

a final feedforward layer configured to receive, as input, the residual connection concatenating the output of the LConv layer with the input to the GLU layer.

9. The computer-implemented method of claim 5 , wherein the operations further comprise:

concatenating the encoded text sequence, the variational embedding, and a reference speaker embedding representing an identity of a reference speaker that uttered the reference audio signal; and

generating the sequence representation based on the duration modeling network receiving, as input, the concatenation of the encoded text sequence, the variational embedding, and the reference speaker embedding.

10. The computer-implemented method of claim 5 , wherein:

the input text sequence includes a sequence of phonemes; and

encoding the input text sequence into the encoded text sequence comprises:

receiving, from a phoneme look-up table, a respective embedding of each phoneme in the sequence of phonemes;

for each phoneme in the sequence of phonemes, processing, using an encoder pre-net neural network of the text encoder, the respective embedding to generate a respective transformed embedding of the phoneme;

processing, using a bank of convolutional blocks, the respective transformed embeddings to generate convolution outputs; and

processing, using a stack of self-attention blocks, the convolution outputs to generate the encoded text sequence.

11. A system for training a non-autoregressive text-to-speech (TTS) model, the system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed by the data processing hardware cause the data processing hardware to perform operations comprising:

obtaining a sequence representation of an encoded text sequence concatenated with a variational embedding;

using a duration model network:

predicting, based on the sequence representation, a phoneme duration for each phoneme represented by the encoded text sequence;

based on the predicted phoneme durations, generating, for each phoneme represented by the encoded text sequence, respective start and end boundaries;

mapping, based on a number of phonemes represented by the encoded text sequence and a number of reference frames in a reference mel-frequency spectrogram sequence, the respective start and end boundaries generated for each phoneme into respective grid matrices;

based on the respective grid matrices mapped from the start and end boundaries, learning, using a first function conditioned on the sequence representation, an interval representation matrix;

determining a product of the interval representation matrix and the sequence representation; and

upsampling, based on the product of the interval representation matrix and the sequence representation, the sequence representation into an upsampled output specifying a number of frames;

generating, as output from a spectrogram decoder comprising a stack of one or more self-attention blocks, based on the upsampled output, one or more predicted mel-frequency spectrogram sequences for the encoded text sequence;

determining a final spectrogram loss based on the one or more predicted mel-frequency spectrogram sequences and the reference mel-frequency spectrogram sequence; and

training the TTS model based on the final spectrogram loss.

12. The system of claim 11 , wherein the first function comprises a respective multi-layer perception-based learnable function.

13. The system of claim 11 , wherein the operations further comprise:

determining a global phoneme duration loss based on the predicted phoneme durations and an average phoneme duration,

wherein training the TTS model is further based on the global phoneme duration loss.

14. The system of claim 13 , wherein training the TTS model based on the final spectrogram loss and the global phoneme duration loss comprises training the duration model network to predict the phoneme duration for each phoneme without using supervised phoneme duration labels extracted from an external aligner.

15. The system of claim 11 , wherein the operations further comprise:

receiving training data including a reference audio signal and a corresponding input text sequence, the reference audio signal comprising a spoken utterance and the input text sequence corresponds to a transcript of the reference audio signal;

encoding, using a residual encoder, the reference audio signal into a variational embedding, the variational embedding disentangling style/prosody information from the reference audio signal; and

encoding, using a text encoder, the input text sequence into the encoded text sequence.

16. The system of claim 15 , wherein:

the residual encoder comprises a global variational autoencoder (VAE); and

encoding the reference audio signal into the variational embedding comprises:

sampling the reference mel-frequency spectrogram sequence from the reference audio signal; and

encoding, using the global VAE, the reference mel-frequency spectrogram sequence into the variational embedding.

17. The system of claim 15 , wherein:

the residual encoder comprises a phoneme-level fine-grained variational autoencoder (VAE); and

encoding the reference audio signal into the variational embedding comprises:

sampling the reference mel-frequency spectrogram sequence from the reference audio signal;

aligning the reference mel-frequency spectrogram sequence with each phoneme in a sequence of phonemes extracted from the input text sequence; and

encoding, using the phoneme-level fine-grained VAE, based on aligning the reference mel-frequency spectrogram sequence with each phoneme in the sequence of phonemes, a sequence of phoneme-level variational embeddings.

18. The system of claim 15 , wherein the residual encoder comprises a stack of lightweight convolution (LConv) blocks, each LConv block in the stack of LConv blocks comprises:

a gated linear unit (GLU) layer;

a LConv layer configured to receive an output of the GLU layer;

a residual connection configured to concatenate an output of the LConv layer with an input to the GLU layer; and

a final feedforward layer configured to receive, as input, the residual connection concatenating the output of the LConv layer with the input to the GLU layer.

19. The system of claim 15 , wherein the operations further comprise:

concatenating the encoded text sequence, the variational embedding, and a reference speaker embedding representing an identity of a reference speaker that uttered the reference audio signal; and

generating the sequence representation based on the duration modeling network receiving, as input, the concatenation of the encoded text sequence, the variational embedding, and the reference speaker embedding.

20. The system of claim 15 , wherein:

the input text sequence includes a sequence of phonemes; and

encoding the input text sequence into the encoded text sequence comprises:

receiving, from a phoneme look-up table, a respective embedding of each phoneme in the sequence of phonemes;

for each phoneme in the sequence of phonemes, processing, using an encoder pre-net neural network of the text encoder, the respective embedding to generate a respective transformed embedding of the phoneme;

processing, using a bank of convolutional blocks, the respective transformed embeddings to generate convolution outputs; and

processing, using a stack of self-attention blocks, the convolution outputs to generate the encoded text sequence.

Assignments (1)

Continuity (3)

References Cited (20)