End-to-end natural and controllable emotional speech synthesis methods
The present disclosure provides acoustic model training methods and systems, and speech synthesis methods and systems. An acoustic model training method may include obtaining a plurality of training samples. Each of the plurality of training samples may include a sample text input, a sample emotion label corresponding to the sample text input, and a sample reference mel spectrum corresponding to the sample text input. The acoustic model training method may include inputting the plurality of training samples into a target model. The target model may include the acoustic model and an auxiliary module. The acoustic model training method may further include iteratively adjusting at least one model parameter of the acoustic model based on a loss target.
1 . A method for training an acoustic model, comprising:
obtaining a plurality of training samples, each of the plurality of training samples including a sample text input, a sample emotion label corresponding to the sample text input, and a sample reference mel spectrum corresponding to the sample text input;
inputting the plurality of training samples into a target model, the target model including the acoustic model and an auxiliary module; and
determining, by the target model, a text sequence vector of the sample text input;
determining, by the target model, a sample prediction style vector by processing the text sequence vector;
iteratively adjusting at least one model parameter of the acoustic model based on a loss target, wherein the sample reference mel spectrum is a ground truth used to determine the loss target, and the sample reference mel spectrum refers to a mel spectrum of a real speech or a standard speech corresponding to the sample text input wherein the sample reference mel spectrum is obtained by:
obtaining a plurality of speech samples of the sample text input, the plurality of speech samples being speech data recorded with various timbres and emotion intensities for the sample text input;
obtaining a candidate mel spectrum of each of the plurality of speech samples;
for each candidate mel spectrum, determining a reference emotion intensity based on the each candidate mel spectrum, the reference emotion intensity referring to a parameter used to evaluate an emotion style and an intensity of an emotion expressed by the speech sample corresponding to the each candidate mel spectrum;
determining a sample emotion intensity corresponding to the sample text input based on the text sequence vector of the sample text input and the sample emotion label corresponding to the sample text input, the sample emotion intensity referring to a parameter used to evaluate an emotion style and an intensity of an emotion expressed by the sample text input; and
selecting, as the sample reference mel spectrum, one of the candidate mel spectrums of the plurality of speech samples based on the sample emotion intensity and the reference emotion intensity of each candidate mel spectrum, the reference emotion intensity of the selected one of the candidate mel spectrums being the same as the sample emotion intensity, or closest to the sample emotion intensity among the reference emotion intensities of the candidate mel spectrums of the plurality of speech samples.
2 . The method of claim 1 , wherein:
the acoustic model comprises:
an encoder configured to determine the text sequence vector of the sample text input; and
an emotion embedding vector determination module configured to determine a sample emotion embedding vector corresponding to the sample emotion label;
the auxiliary module comprises:
an unsupervised module configured to determine a sample reference style vector corresponding to the sample reference mel spectrum.
3 . The method of claim 2 , wherein the acoustic model further comprises:
a vector processing module configured to determine a comprehensive emotion vector based on a sum of the sample emotion embedding vector and the sample reference style vector, wherein the comprehensive emotion vector is a character-level embedding vector.
4 . The method of claim 3 , wherein the acoustic model further comprises:
a decoder configured to determine a sample prediction mel spectrum based on a cascade vector of the text sequence vector and the comprehensive emotion vector.
5 . The method of claim 4 , wherein
the vector processing module is further configured to determine a hidden state vector that refers to a low-dimensional dense embedding vector related to the comprehensive emotion vector, and
the auxiliary module further comprises an emotion classifier configured to determine a vector emotion category based on the hidden state vector.
6 . The method of claim 4 , wherein
the auxiliary module further comprises an emotion identification module configured to determine a prediction deep emotion feature corresponding to the sample prediction mel spectrum and a reference deep emotion feature corresponding to the sample reference mel spectrum, and the loss target includes a difference loss between the prediction deep emotion feature and the reference deep emotion feature; or
the loss target includes a difference loss between the sample prediction mel spectrum and the sample reference mel spectrum.
7 . A system for training an acoustic model, comprising:
at least one computer-readable storage medium including a set of instructions; and
at least one processing device communicating with the computer-readable storage medium, wherein when executing the set of instructions, the at least one processing device is configured to:
obtain a plurality of training samples, each of the plurality of training samples including a sample text input, a sample emotion label corresponding to the sample text input, and a sample reference mel spectrum corresponding to the sample text input;
input the plurality of training samples into a target model, the target model including the acoustic model and an auxiliary module;
determine, by the target model, a sample reference style vector corresponding to the sample reference mel spectrum, including:
segmenting the sample reference mel spectrum into sub-segment sample reference mel spectrums each of which corresponds to a sub-segment sample text input of the sample text input;
determining a sub-segment sample reference style vector of each sub-segment sample reference mel spectrum by processing the each sub-segment sample reference mel spectrum, the sub-segment sample reference style vector including a sub-segment sample reference style feature and a sample reference style relationship feature, the sub-segment sample reference style feature referring to a vector representing an emotion style of the sub-segment sample text input corresponding to the each sub-segment sample reference mel spectrum, the sample reference style relationship feature referring to a vector representing a relationship between the sub-segment sample text input and the sub-segment sample reference style vectors of previous and subsequent sub-segment sample text inputs; and
obtaining the sample reference style vector by arranging the sub-segment sample reference style vectors in order; and
iteratively adjust at least one model parameter of the acoustic model based on a loss target, the loss target being determined based on the sample reference style vector.
8 . The system of claim 7 , wherein: the acoustic model comprises:
an encoder configured to determine a text sequence vector of the sample text input; and
an emotion embedding vector determination module configured to determine a ample emotion embedding vector corresponding to the sample emotion label;
a vector processing module configured to determine a comprehensive emotion vector based on a sum of the sample emotion embedding vector and the sample reference style vector, wherein the comprehensive emotion vector is a character-level embedding vector;
a decoder configured to determine a sample prediction mel spectrum based on a cascade vector of the text sequence vector and the comprehensive emotion vector;
the auxiliary module comprises:
an unsupervised module configured to determine the sample reference style vector corresponding to the sample reference mel spectrum.
9 . The system of claim 8 , wherein
the vector processing module is further configured to determine a hidden state vector,
the acoustic model further comprises a vector prediction module configured to determine a sample prediction style vector based on the text sequence vector, and
the auxiliary module further comprises
an emotion classifier configured to determine a vector emotion category based on the hidden state vector, and
an emotion identification module configured to determine a prediction deep emotion feature corresponding to the sample prediction mel spectrum and a reference deep emotion feature corresponding to the sample reference mel spectrum.
10 . The system of claim 9 , wherein the loss target comprises at least one of the following:
a difference loss between the sample prediction style vector and the sample reference style vector;
a classification loss of the vector emotion category;
a difference loss between the sample prediction mel spectrum and the sample reference mel spectrum; or
a difference loss between the prediction deep emotion feature and the reference deep emotion feature.
11 . A speech synthesis method, comprising:
obtaining a text input and an emotion label corresponding to the text input;
generating, by an acoustic model, a prediction mel spectrum corresponding to the text input based on the text input and the emotion label, and
generating a prediction speech corresponding to the text input based on the prediction mel spectrum, wherein
the acoustic model has been trained, and a training of the acoustic model comprises:
obtaining a plurality of training samples, each of the plurality of training samples including a sample text input, a sample emotion label corresponding to the sample text input, and a sample reference mel spectrum corresponding to the sample text input;
inputting the plurality of training samples into a target model, the target model including the acoustic model and an auxiliary module; and
iteratively adjusting at least one model parameter of the acoustic model based on a loss target, the sample reference mel spectrum being a ground truth used to determine the loss target;
the acoustic model comprises:
an encoder configured to determine a text sequence vector of the sample text input;
an emotion embedding vector determination module configured to determine a sample emotion embedding vector corresponding to the sample emotion label;
a vector processing module configured to determine a comprehensive emotion vector based on a sum of the sample emotion embedding vector and a sample reference style vector; and
the auxiliary module comprises:
an unsupervised module configured to determine the sample reference style vector corresponding to the sample reference mel spectrum;
wherein
the vector processing module is further configured to determine a hidden state vector, the hidden state vector referring to a low-dimensional dense embedding vector related to the comprehensive emotion vector;
the auxiliary module further comprises an emotion classifier configured to determine a vector emotion category based on the hidden state vector; and
a difference loss between the vector emotion category and the sample emotion label is a loss term of the loss target.
12 . The speech synthesis method of claim 11 , further comprising:
generating an actual text sequence vector corresponding to the text input based on the text input;
generating an actual emotion embedding vector corresponding to the emotion label based on the emotion label:
generating a prediction style vector corresponding to the text input by processing the actual text sequence vector; and
generating the prediction mel spectrum corresponding to the text input based on the actual text sequence vector, the actual emotion embedding vector, and the prediction style vector;
wherein
the acoustic model further comprises a vector prediction module, and
the generating, by the acoustic model, the prediction mel spectrum corresponding to the text input based on the text input and the emotion label comprises:
determining, by the vector processing module, an actual comprehensive emotion vector corresponding to the text input based on the actual text sequence vector and a sum of the prediction style vector and the actual emotion embedding vector; and
generating, by the decoder, the prediction mel spectrum corresponding to the text input based on an actual cascade vector of the actual comprehensive emotion vector and the actual text sequence vector.
13 . The speech synthesis method of claim 11 , wherein the acoustic model further comprises:
a decoder configured to determine a sample prediction mel spectrum based on a cascade vector of the text sequence vector and the comprehensive emotion vector:
a vector prediction module configured to determine a sample prediction style vector based on the text sequence vector; and
an emotion identification module configured to determine a prediction deep emotion feature corresponding to the sample prediction mel spectrum and a reference deep emotion feature corresponding to the sample reference mel spectrum.
14 . The speech synthesis method of claim 13 , wherein the loss target is a sum of the difference loss between the vector emotion category and the sample emotion label, and at least one of the following:
a difference loss between the sample prediction style vector and the sample reference style vector;
a difference loss between the sample prediction mel spectrum and the sample reference mel spectrum; or
a difference loss of the prediction deep emotion feature and the reference deep emotion feature.
15 . The method of claim 5 , wherein the loss target further includes a difference loss between the vector emotion category and the sample emotion label.
16 . The method of claim 6 , wherein the emotion identification module includes at least one of a bidirectional gated recurrent unit, a pooling layer, or a linear layer, and a feature of a preset dimension after the pooling layer is used as a deep feature.
17 . The method of claim 1 , wherein the auxiliary module includes:
an emotion intensity extraction module that is a machine learning model configured to determine the sample emotion intensity; and
a style identification module that is a machine learning model configured to determine the reference emotion intensity.
18 . The system of claim 10 , wherein the classification loss is a difference loss between the vector emotion category and the sample emotion label.
19 . The system of claim 18 , wherein the difference loss between the vector emotion category and the sample emotion label is a loss term of the loss target.
20 . The system of claim 19 , wherein the loss target is a sum of the difference loss between the vector emotion category and the sample emotion label, and at least one of the following:
the difference loss between the sample prediction style vector and the sample reference style vector;
the difference loss between the sample prediction mel spectrum and the sample reference mel spectrum; or
the difference loss of the prediction deep emotion feature and the reference deep emotion feature.