Clockwork hierarchical variational encoder

1. A method comprising:

receiving, at data processing hardware, a text utterance having at least one word, each word having at least one syllable, each syllable having at least one phoneme;

selecting, by the data processing hardware, a mel spectral embedding for the text utterance; and

for each phoneme, using the selected mel spectral embedding:

predicting a duration of the corresponding phoneme by encoding linguistic features of the corresponding phoneme with a corresponding syllable embedding for the syllable that includes the corresponding phoneme; and

generating a plurality of fixed-length predicted mel-frequency spectrogram frames based on the predicted duration for the corresponding phoneme, each fixed-length predicted mel-frequency spectrogram frame representing mel-spectral information of the corresponding phoneme.

2. The method of claim 1 , wherein a network representing a hierarchical linguistic structure of the text utterance comprises:

a first level including each syllable of the text utterance;

a second level including each phoneme of the text utterance; and

a third level including each fixed-length predicted mel-frequency spectrogram frame.

3. The method of claim 2 , wherein:

the first level of the network representing the hierarchical linguistic structure comprises a long short-term memory (LSTM) processing cell representing each syllable of the text utterance;

the second level of the network representing the hierarchical linguistic structure comprises a LSTM processing cell representing each phoneme of the text utterance, the LSTM processing cells of the second level clocking relative to and faster than the LSTM processing cells of the first level; and

the third level of the network representing the hierarchical linguistic structure comprises a LSTM processing cell representing each fixed-length predicted mel-frequency spectrogram frame, the LSTM processing cells of the third level clocking relative to and faster than the LSTM processing cells of the second level.

4. The method of claim 1 , further comprising converting, by the data processing hardware, using a vocoder network, the plurality of generated fixed-length predicted mel-frequency spectrogram frames into a time-domain audio waveform.

5. The method of claim 4 , wherein the vocoder network is conditioned on mel-frequency spectrograms for conversion into time-domain audio waveforms.

6. The method of claim 4 , wherein the vocoder network and a deep neural network representing a hierarchical linguistic structure of the text utterance are trained separately.

7. The method of claim 1 , further comprising, for each phoneme, using the selected mel spectral embedding:

predicting, by the data processing hardware, an energy contour of the corresponding phoneme based on the predicted duration for the phoneme; and

generating, by the data processing hardware, a plurality of fixed-length predicted energy frames based on the predicted duration for the corresponding phoneme, each fixed-length energy frame representing the predicted energy contour of the corresponding phoneme.

8. The method of claim 7 , wherein a hierarchical linguistic structure represents the text utterance, the hierarchical linguistic structure comprising:

a first level including each syllable of the text utterance;

a second level including each phoneme of the text utterance;

a third level including each fixed-length predicted mel-frequency spectrogram frame; and

a fourth level parallel to the third level and including each fixed-length predicted energy frame for each phoneme of the text utterance.

9. The method of claim 1 , further comprising, for each syllable, using the selected mel spectral embedding:

predicting, by the data processing hardware, a duration of the syllable by summing the predicted durations for each phoneme associated with the syllable;

predicting, by the data processing hardware, a pitch contour of the syllable based on the predicted duration for the syllable; and

generating, by the data processing hardware, a plurality of fixed-length predicted pitch frames based on the predicted duration of the syllable, each fixed-length predicted pitch frame representing part of the predicted pitch contour of the syllable.

10. The method of claim 9 , wherein a network representing a hierarchical linguistic structure of the text utterance comprises:

a first level including each syllable of the text utterance;

a second level including each phoneme of the text utterance;

a third level including each fixed-length predicted mel-frequency spectrogram frame; and

a fourth level parallel to the third level and including each fixed-length predicted pitch frame for each syllable of the text utterance.

11. The method of claim 1 , wherein the mel spectral embedding comprises a fixed-length numerical vector.

12. The method of claim 1 , further comprising:

receiving, by the data processing hardware, training data including a plurality of reference audio signals, each reference audio signal comprising a spoken utterance of human speech and having a corresponding prosody; and

training, by the data processing hardware, a deep neural network for a mel-frequency spectrogram model by:

encoding each reference audio signal into a corresponding fixed-length utterance embedding representing the corresponding prosody of the reference audio signal; and

decoding each corresponding fixed-length utterance embedding into a plurality of corresponding fixed-length mel-frequency spectrogram frames representing the corresponding prosody of the reference audio signal.

13. A system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:

receiving a text utterance having at least one word, each word having at least one syllable, each syllable having at least one phoneme;

selecting a mel spectral embedding for the text utterance; and

for each phoneme, using the selected mel spectral embedding:

predicting a duration of the corresponding phoneme by encoding linguistic features of the corresponding phoneme with a corresponding syllable embedding for the syllable that includes the corresponding phoneme; and

generating a plurality of fixed-length predicted mel-frequency spectrogram frames based on the predicted duration for the corresponding phoneme, each fixed-length predicted mel-frequency spectrogram frame representing mel-spectral information of the corresponding phoneme.

14. The system of claim 13 , wherein a network representing a hierarchical linguistic structure of the text utterance comprises:

a first level including each syllable of the text utterance;

a second level including each phoneme of the text utterance; and

a third level including each fixed-length predicted mel-frequency spectrogram frame.

15. The system of claim 14 , wherein:

the first level of the network representing the hierarchical linguistic structure comprises a long short-term memory (LSTM) processing cell representing each syllable of the text utterance;

the second level of the network representing the hierarchical linguistic structure comprises a LSTM processing cell representing each phoneme of the text utterance, the LSTM processing cells of the second level clocking relative to and faster than the LSTM processing cells of the first level; and

the third level of the network representing the hierarchical linguistic structure comprises a LSTM processing cell representing each fixed-length predicted mel-frequency spectrogram frame, the LSTM processing cells of the third level clocking relative to and faster than the LSTM processing cells of the second level.

16. The system of claim 14 , wherein the operations further comprise, for each syllable, using the selected mel spectral embedding:

predicting a duration of the syllable by summing the predicted durations for each phoneme associated with the syllable;

predicting a pitch contour of the syllable based on the predicted duration for the syllable; and

generating a plurality of fixed-length predicted pitch frames based on the predicted duration of the syllable, each fixed-length predicted pitch frame representing part of the predicted pitch contour of the syllable.

17. The system of claim 16 , wherein a network representing a hierarchical linguistic structure of the text utterance comprises:

a first level including each syllable of the text utterance;

a second level including each phoneme of the text utterance;

a third level including each fixed-length predicted mel-frequency spectrogram frame; and

a fourth level parallel to the third level and including each fixed-length predicted pitch frame for each syllable of the text utterance.

18. The system of claim 14 , wherein the mel spectral embedding comprises a fixed-length numerical vector.

19. The system of claim 14 , wherein the operations further comprise:

receiving training data including a plurality of reference audio signals, each reference audio signal comprising a spoken utterance of human speech and having a corresponding prosody; and

training a deep neural network for a mel-frequency spectrogram model by:

encoding each reference audio signal into a corresponding fixed-length utterance embedding representing the corresponding prosody of the reference audio signal; and

decoding each corresponding fixed-length utterance embedding into a plurality of corresponding fixed-length mel-frequency spectrogram frames representing the corresponding prosody of the reference audio signal.

20. The system of claim 13 , wherein the operations further comprise converting, using a vocoder network, the plurality of generated fixed-length predicted mel-frequency spectrogram frames into a time-domain audio waveform.

21. The system of claim 20 , wherein the vocoder network is conditioned on mel-frequency spectrograms for conversion into time-domain audio waveforms.

22. The system of claim 20 , wherein the vocoder network and a deep neural network representing a hierarchical linguistic structure of the text utterance are trained separately.

23. The system of claim 13 , wherein the operations further comprise, for each phoneme, using the selected mel spectral embedding:

predicting an energy contour of the corresponding phoneme based on the predicted duration for the phoneme; and

generating a plurality of fixed-length predicted energy frames based on the predicted duration for the corresponding phoneme, each fixed-length energy frame representing the predicted energy contour of the corresponding phoneme.

24. The system of claim 23 , wherein a hierarchical linguistic structure represents the text utterance, the hierarchical linguistic structure comprising:

a first level including each syllable of the text utterance;

a second level including each phoneme of the text utterance;

a third level including each fixed-length predicted mel-frequency spectrogram frame; and

a fourth level parallel to the third level and including each fixed-length predicted energy frame for each phoneme of the text utterance.