Method and apparatus for training encoder
View Patent ↗A method and an apparatus for training a speech transformation model are provided. The method and the apparatus are capable of generating a natural speech suitable for context and improving accuracy of pronunciation by training the first encoder (e.g., a encoder of the flow-based model) and the second encoder (e.g., a encoder of the Tacotron 2 model) in parallel.
1 . A computer implement method for training a speech transformation model, which includes a first encoder, a second encoder, a third encoder, one or more attention modules, and a phoneme duration prediction unit, using dataset including text and speech, the method comprising:
generating, using the first encoder, first output data from training data generated by preprocessing the text, the first output data related to first features of the text;
generating, using the second encoder, second output data from the training data generated by preprocessing the text, the second output data related to second features of the text;
generating, using the third encoder, third output data from the speech corresponding to the text, the third output data related to features of the speech;
generating, using the one or more attention modules, (i) a first similarity from the first output data and the third output data and (ii) a second similarity from the second output data and the third output data, the first similarity and the second similarity representing similarities between the first and second features of the text and the features of the speech;
generating, using the phoneme duration prediction unit, a predicted phoneme duration of the text based on a weighted sum of the first output data with a weight based on the first similarity and the second output data with a weight based on the second similarity; and
updating one or more parameters of the speech transformation model based on a loss function to which the at least one of the first output data, the second output data, or the third output data are applied.
2 . The method of claim 1 , further comprising:
refining the speech transformation model using a dataset of a language not used for the training of the speech transformation model.
3 . The method of claim 1 , wherein the first encoder is an encoder of a transformer model configured to perform text-to-speech conversion, and the second encoder is an encoder of Tacotron 2 model configured to perform text-to-speech conversion.
4 . The method of claim 1 , wherein the first encoder and the second encoder are configured to generate the first output data and the second output data based on style-adaptive layer normalization (SALN).
5 . The method of claim 1 , wherein the similarity is related to information regarding alignment of the text and the speech.
6 . The method of claim 1 , wherein generating the first output data and the second output data comprises:
extracting text features from the training data, and
performing language embedding indicating received language on the extracted text features for output data generation.
7 . The method of claim 1 , wherein the training data is obtained by performing at least one of one-hot encoding, character embedding, or position encoding to a dataset.
8 . The method of claim 1 , wherein the speech transformation model further includes a decoder configured to generate a synthetic speech from the third output data; and
wherein the updating includes updating one or more parameters of the speech transformation model further based on a loss between the speech of the training data and the synthetic speech.
9 . The method of claim 1 , wherein the first encoder and the second encoder are configured to generate the first output data and the second output data based on style-adaptive layer normalization (SALN) during the updating, wherein the method further comprises:
replacing the SALN of the first encoder and the second encoder with layer normalization after the updating; and
refining the speech transformation model using a dataset of a language different from languages used for the updating.
10 . An apparatus for training a speech transformation model including a first encoder, a second encoder, a third encoder, one or more attention modules, and a phoneme duration prediction unit, the apparatus comprising:
a memory storing instructions; and
at least one or more processors configured to execute the instructions to perform operations comprising:
generating, using the first encoder, first output data from training data generated by preprocessing a text, the first output data related to first features of the text;
generating, using the second encoder, second output data from the training data generated by preprocessing the text of the training data, the second output data related to second features of the text;
generating, using the third encoder, third output data from speech corresponding to the text, the third output data related to features of the speech;
generating, using the one or attention modules, (i) a first similarity from the first output data and the third output data and (ii) a second similarity from the second output data and the third output data, the first similarity and the second similarity representing similarities between the first and second features of the text and the features of the speech;
generating, using the phoneme duration prediction unit, a predicted phoneme duration of the text based on a weighted sum of the first output data with a weight based on the first similarity and the second output data with a weight based on the second similarity; and
updating one or more parameters of the speech transformation model based on a loss function to which the at least one of the first output data, the second output data, or the third output data are applied.
11 . The apparatus of claim 10 , wherein the operations further comprise refining the speech transformation model using a dataset of a language not used for the training of the speech transformation model.
12 . The apparatus of claim 10 , wherein the first encoder is an encoder of a transformer model configured to perform text-to-speech conversion, and the second encoder is an encoder of Tacotron 2 model configured to perform text-to-speech conversion.
13 . The apparatus of claim 10 , wherein the first encoder and the second encoder are configured to generate the first output data and the second output data based on style-adaptive layer normalization (SALN).
14 . The apparatus of claim 10 , wherein the similarity is related to information regarding alignment of the text and the speech.
15 . The apparatus of claim 10 , wherein generating the first output data and the second output data include;
extracting text features form the training data; and
performing language embedding indicating received language on the extracted text features for output data generation.
16 . The apparatus of claim 10 , wherein the training data is obtained by performing at least one of one-hot encoding, character embedding, or position encoding to a dataset.
17 . The apparatus of claim 10 , wherein the speech transformation model further includes a decoder configured to generate a synthetic speech from the third output data; and
wherein the updating includes updating one or more parameters of the speech transformation model further based on a loss between the speech of the training data and the synthetic speech.
18 . The apparatus of claim 10 , wherein the first encoder and the second encoder are configured to generate the first output data and the second output data based on style-adaptive layer normalization (SALN) during the updating
wherein the operations further comprise:
replacing the SALN of the first encoder and the second encoder with layer normalization after the updating; and
refining the speech transformation model using a data set of a language different from languages used for updating.