Text-to-speech (TTS) processing
During text-to-speech processing, a speech model creates output audio data, including speech, that corresponds to input text data that includes a representation of the speech. A spectrogram estimator estimates a frequency spectrogram of the speech; the corresponding frequency-spectrogram data is used to condition the speech model. A plurality of acoustic features corresponding to different segments of the input text data, such as phonemes, syllable-level features, and/or word-level features, may be separately encoded into context vectors; the spectrogram estimator uses these separate context vectors to create the frequency spectrogram.
1. A computer-implemented method comprising:
receiving input audio data representing an utterance corresponding to a request to create requested synthesized speech;
processing the input audio data using a first component to determine first acoustic-feature data corresponding to at least one emotion represented in the utterance;
determining first data representing words corresponding to the requested synthesized speech;
processing the first data to determine second acoustic-feature data;
processing the first acoustic-feature data and the second acoustic-feature data to determine spectrogram data; and
processing the spectrogram data to determine output audio data representing synthesized speech of the words, the synthesized speech reflecting the at least one emotion.
2. The computer-implemented method of claim 1 , further comprising:
processing the input audio data to determine the first data representing the words.
3. The computer-implemented method of claim 2 , wherein:
the first component comprises a first encoder; and
processing the input audio data to determine the first data comprises processing the input audio data using a second encoder to determine the first data.
4. The computer-implemented method of claim 1 , wherein processing the first data and the first acoustic-feature data to determine output audio data comprises using at least one model comprising at least one hidden layer to determine the output audio data.
5. The computer-implemented method of claim 1 , further comprising:
processing the spectrogram data with a first model to determine model output data; and
processing the model output data and the spectrogram data using a second model to determine output data,
wherein the output data is used to determine the output audio data.
6. The computer-implemented method of claim 1 , wherein:
the first data corresponds to a first time resolution; and
the first acoustic-feature data corresponds to a second time resolution different from the first time resolution.
7. The computer-implemented method of claim 1 , wherein:
the output audio data comprises a first portion corresponding to a first portion of the words and a second portion corresponding to a second portion of the words;
the emotion corresponds to a fearful emotion;
the method further comprises determining that the emotion corresponds to the first portion of the words; and
the first portion of the output audio data comprises higher frequency audio data than the second portion of the output audio data.
8. The computer-implemented method of claim 1 , wherein processing the first data and the first acoustic-feature data to determine the output audio data comprises:
processing the first acoustic-feature data using an attention network to determine modified first acoustic-feature data; and
processing the first data and the modified first acoustic-feature data to determine the output audio data.
9. A system comprising:
at least one processor; and
at least one memory comprising instructions that, when executed by the at least one processor, cause the system to:
receive input audio data representing an utterance corresponding to a request to create requested synthesized speech;
process the input audio data using a first component to determine first acoustic-feature data corresponding to at least one emotion represented in the utterance;
determine first data representing words corresponding to the requested synthesized speech;
process the first data to determine second acoustic-feature data;
process the first acoustic-feature data and the second acoustic-feature data to determine spectrogram data; and
process the spectrogram data to determine output audio data representing synthesized speech of the words, the synthesized speech reflecting the at least one emotion.
10. The system of claim 9 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
process the input audio data to determine the first data representing the words.
11. The system of claim 10 , wherein:
the first component comprises a first encoder; and
the instructions that cause the system to process the input audio data to determine the first data comprise instructions that, when executed by the at least one processor, cause the system to process the input audio data using a second encoder to determine the first data.
12. The system of claim 9 , wherein the instructions that cause the system to process the input audio data to process the first data and the first acoustic-feature data to determine output audio data comprise instructions that, when executed by the at least one processor, cause the system to use at least one model comprising at least one hidden layer to determine the output audio data.
13. The system of claim 9 , wherein the instructions that cause the system to process the input audio data to process the first data and the first acoustic-feature data to determine the output audio data comprise instructions that, when executed by the at least one processor, cause the system to:
process the first data to determine second acoustic-feature data; and
process the first acoustic-feature data and the second acoustic-feature data to determine the output audio data.
14. The system of claim 9 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
process the spectrogram data with a first model to determine model output data; and
process the model output data and the spectrogram data using a second model to determine output data,
wherein the output data is used to determine the output audio data.
15. The system of claim 9 , wherein:
the output audio data comprises a first portion corresponding to a first portion of the words and a second portion corresponding to a second portion of the words;
the emotion corresponds to a fearful emotion;
the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to determine that the emotion corresponds to the first portion of the words; and
the first portion of the output audio data comprises higher frequency audio data than the second portion of the output audio data.
16. The system of claim 9 , wherein the instructions that cause the system to process the input audio data to process the first data and the first acoustic-feature data to determine the output audio data comprise instructions that, when executed by the at least one processor, cause the system to:
process the first acoustic-feature data using an attention network to determine modified first acoustic-feature data; and
process the first data and the modified first acoustic-feature data to determine the output audio data.
17. The system of claim 9 , wherein:
the first data corresponds to a first time resolution; and
the first acoustic-feature data corresponds to a second time resolution different from the first time resolution.