Text-to-speech (TTS) processing
A speech model is trained using multi-task learning. A first task may correspond to how well predicted audio matches training audio; a second task may correspond to a metric of perceived audio quality. The speech model may include, during training, layers related to the second task that are discarded at runtime.
1. A computer-implemented method for generating speech from text, the method comprising:
training, using multi-task learning, to create a speech model, wherein:
the speech model includes:
a sample model configured to input text data and output audio samples,
a conditioning model configured to input text metadata corresponding to the input text data and to condition the sample model, and
an output model configured to input the output audio samples and output audio output data, and
the training includes:
using a first section of the conditioning model, configuring at least one hidden layer of the sample model in accordance with a first task, wherein the first task includes minimizing a difference between the audio output data and corresponding training data;
using a second section of the conditioning model, configuring the at least one hidden layer of the sample model in accordance with a second task, wherein the second task includes maximizing a metric of perceived quality of the audio output data;
including the first section of the conditioning model in the speech model, and
discarding the second section of the conditioning model; and
generating, using first text data and the speech model, first audio output data corresponding to the input text data.
2. The computer-implemented method of claim 1 , wherein maximizing the metric of perceived quality comprises at least one of:
computing, using the audio output data, a perceptual evaluation of speech quality (PESQ) standard; and
computing, using the audio output data, a mel-frequency cepstrum (WC).
3. The computer-implemented method of claim 1 , further comprising:
selecting a layer of the output model for training the at least one hidden layer of the sample model;
generating, during configuring the at least one hidden layer of the sample model in accordance with the second task using the layer of the output model, output feedback data; and
training the at least one hidden layer of the sample model with the output feedback data.
4. The computer-implemented method of claim 1 , further comprising, during the training, removing one or more nodes to create the hidden layer,
wherein configuring the at least one hidden layer of the sample model in accordance with the second task comprises configuring unremoved nodes in the hidden layer.
5. A computer-implemented method comprising:
receiving text data;
receiving text metadata corresponding to the text data; and
generating, using the text data, the text metadata, and a speech model, first audio output data corresponding to the text data,
wherein the speech model includes:
a hidden layer having values determined at least in part by:
a first task using a first section of a model, and
a second task using a second section of the model, wherein the second task includes maximizing a metric of perceived quality of the first audio output data,
the first section of the model, and
wherein the speech model does not include the second section of the model.
6. The computer-implemented method of claim 5 , wherein the first section of the model corresponds to a first section of a conditioning model, and wherein the second section of the model corresponds to a second section of the conditioning model.
7. The computer-implemented method of claim 5 ,
wherein the first section of the model corresponds to a first section of an output model, and
wherein the second section of the model corresponds to a second section of the output model.
8. The computer-implemented method of claim 5 , wherein the first task comprises measuring an accuracy of the speech model, and further comprising comparing the first audio output data to corresponding audio training data.
9. The computer-implemented method of claim 5 , further comprising:
training, using a section of a second model, the at least one hidden layer of the speech model in accordance with the second task.
10. The computer-implemented method of claim 5 , wherein maximizing the metric of perceived quality comprises at least one of:
computing, using the first audio output data, a perceptual evaluation of speech quality (PESQ) standard; and
computing, using the first audio output data, a mel-frequency cepstrum (MFC).
11. The computer-implemented method of claim 5 , further comprising, during maximizing the metric of perceived quality of the first audio output data, reducing a number of nodes of the hidden layer.
12. The computer-implemented method of claim 11 , further comprising determining a second speech model using an output of the hidden layer.
13. A system comprising:
at least one processor; and
at least one memory including instructions that, when executed by the at least one processor, cause the system to:
receive text data;
receive text metadata corresponding to the text data; and
generate, using the text data, the text metadata, and a speech model, first audio output data corresponding to the text data,
wherein the speech model includes:
a hidden layer having values determined at least in part by:
a first task, using a first section of a model, and
a second task, using a second section of the model, wherein the second task includes maximizing a metric of perceived quality of the first audio output data,
the first section of the model, and
wherein the speech model does not include the second section of the model.
14. The system of claim 13 ,
wherein the first section of the model corresponds to a first section of a conditioning model, and
wherein the second section of the model corresponds to a second section of the conditioning model.
15. The system of claim 13 ,
wherein the first section of the model corresponds to a first section of an output model,
and wherein the second section of the model corresponds to a second section of the output model.
16. The system of claim 13 , wherein the first task comprises measuring an accuracy of the speech model, and wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to compare the first audio output data to corresponding audio training data.
17. The system of claim 13 , wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to train, using a section of a second model, the at least one hidden layer of the speech model in accordance with the second task.
18. The system of claim 13 , wherein the instructions that cause the system to maximize the metric of perceived quality further cause the system to:
compute, using the first audio output data, a perceptual evaluation of speech quality (PESQ) standard; or
compute, using the first audio output data, a mel-frequency cepstrum (WC).
19. The system of claim 13 , wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to, during maximizing the metric of perceived quality of the first audio output data, reduce a number of nodes of the hidden layer.
20. The system of claim 13 , wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to determine a second speech model using an output of the hidden layer.