Speech synthesis apparatus and method thereof
Disclosed is a speech synthesis method including acquiring second speech data and a target text, acquiring first information includes embedding information corresponding to the second speech data, acquiring second information including embedding information of the second speech data, the embedding information in relation with components generated based on a sequence of the target text, and acquiring audio data corresponding to the target text and reflecting characteristics of speech of a speaker based on the first information and the second information.
1. A speech synthesis method, the speech synthesis method comprising:
acquiring a second set of speech data and a target text;
acquiring, using a text-to-speech synthesis model trained based on text data corresponding to a first set of speech data and at least a portion of the first set of speech data, a first set of information, wherein the first set of information includes a first set of embedding information comprising the second set of speech data;
acquiring, using the text-to-speech synthesis model, a second set of information, wherein:
the second set of information includes a second set of embedding information, comprising embeddings of the second set of speech data,
the second set of embedding information is acquired by deploying an attention mechanism using query components generated based on a sequence of the target text, and
acquiring the second set of information comprises:
encoding the target text, and
extracting the query components from the encoded target text. wherein:
the query components are generated based on a sequence of the encoded target text, and
the sequence is generated based on the first set of information and the encoded target text;
acquiring audio data, using the text-to-speech synthesis model, wherein the audio data:
corresponds to the target text, and
reflects characteristics of speech of a speaker of the second set of speech data, as a sound spectrum visualization generated based on the first set of information and the second set of information; and
deriving, using the text-to-speech synthesis model, a speech recording corresponding to the audio data.
2. The speech synthesis method of claim 1 , wherein:
the first set of embedding information includes a vector that:
corresponds to the second set of speech data, and
has a predetermined length independent of a length of the second set of speech data; and
the second set of embedding information includes a plurality of vectors, with variable lengths.
3. The speech synthesis method of claim 2 , wherein lengths of the plurality of vectors are variably determined based on a length of the target text.
4. The speech synthesis method of claim 1 , wherein the extracting of the query components comprises:
inputting the encoded target text into a first model, wherein the first model is included in the text-to-speech synthesis model; and
concatenating an output of the first model with the first set of information, wherein the output of the first model includes a plurality of preliminary components generated based on the sequence.
5. The speech synthesis method of claim 4 , wherein the concatenating of the output of the first model with the first set of information comprises:
copying the first set of information based on a number of the plurality of preliminary components included in the output of the first model; and
concatenating the copied first set of information with the plurality of preliminary components.
6. The speech synthesis method of claim 1 , wherein the acquiring of the second set of information further comprises inputting a result, obtained by extracting the query components generated based on the sequence of the encoded target text, into a first neural network included in the text-to-speech synthesis model.
7. The speech synthesis method of claim 1 , wherein the acquiring of the second set of information further comprises:
inputting the second set of speech data to a second neural network included in the text-to-speech synthesis model;
inputting the second set of speech data to a third neural network included in the text-to-speech synthesis model; and
acquiring the second set of information based on an output of the second neural network, an output of the third neural network, and the query components generated based on the sequence of the encoded target text.
8. The speech synthesis method of claim 1 , wherein:
the acquiring of the second set of information further comprises a plurality of detailed operations, and
a j-th detailed operation included in the plurality of detailed operations corresponds to acquiring a j-th component of the second set of information based on the encoded target text and the first set of information.
9. The speech synthesis method of claim 8 , wherein the j-th detailed operation corresponds to acquiring the j-th component of the second set of information based on Equation 1, wherein Equation 1 follows form:
e
v
j
=
softmax
(
f
(
Q
j
)
f
(
K
)
T
d
m
)
f
(
V
)
,
wherein:
e v j is the j-th component of the second set of information,
Q j is a result of inputting a result obtained by extracting a j-th query component of the query components generated based on the sequence of the encoded target text, into a first neural network included in the text-to-speech synthesis model,
K is an output of a second neural network included in the text-to-speech synthesis model when the second set of speech data is input into the second neural network,
V is an output of a third neural network included in the text-to-speech synthesis model when the second set of speech data is input into the third neural network,
d m is a dimension of the query components generated based on the sequence of the encoded target text,
f is a flattening function, and
softmax is a softmax function.
10. The speech synthesis method of claim 1 , wherein the acquiring of the audio data comprises:
acquiring a third set of information based on a result obtained by inputting the second set of information and the query components according to the sequence of the second set of speech data to a first neural network included in the text-to-speech synthesis model;
inputting the third set of information into a fourth neural network included in the text-to-speech synthesis model; and
acquiring audio data corresponding to the third set of information based on an output of the fourth neural network.
11. The speech synthesis method of claim 1 , wherein:
the text-to-speech synthesis model comprises:
a first model configured to:
receive text information including at least one of the target text or the text data,
encode the text information, and
output a result of the encoding;
a second model configured to:
receive at least one of the first set of speech data or the second set of speech data, and
output a third set of embedding information, comprising the speech data received by the second model;
a third model configured to:
receive at least one of the first set of speech data or the second set of speech data, and
output a fourth set of embedding information comprising embeddings of the speech data received by the third model, wherein the fourth set of embedding information is acquired by deploying the attention mechanism using the query components generated based on the sequence of the text information; and
a fourth model configured to:
acquire alignment information on an alignment between the text information and the speech data received by the second model by receiving the output of the first model and the output of the second model, and
output a result of inputting, into a neural network:
the fourth set of embedding information, and
the alignment information on the alignment, and
the result output from the fourth model is input to the third model so that the third model is configured to run in association with the fourth model.
12. The speech synthesis method of claim 1 , wherein the text-to-speech synthesis model may be trained by:
acquiring the first set of speech data and the text data;
encoding the text data;
acquiring a fourth set of information including alignment information on an alignment in time between the first set of speech data and the text data, wherein the alignment is based on a third set of embedding information, comprising the first set of speech data, and the encoded text data;
acquiring a fifth set of information including a sixth set of embedding information, wherein the sixth set of embedding information comprises embeddings of the first set of speech data and is acquired by deploying the attention mechanism using query components generated based on the sequence of the text data;
acquiring a sixth set of information based on the fourth set of information and the fifth set of information;
inputting the sixth set of information into a fourth neural network included in the text-to-speech synthesis model; and
training the text-to-speech synthesis model on parameter information based on an output of the fourth neural network.
13. The speech synthesis method of claim 12 , wherein:
the parameter information includes at least one piece of weight information and at least one piece of bias information associated with the text-to-speech synthesis model,
the at least one piece of weight information includes weight information associated with at least a portion of each of a first model, a second model, a third model, and a fourth model included in the text-to-speech synthesis model,
the at least one piece of bias information includes bias information associated with at least a portion of each of the first model, the second model, the third model, and the fourth model, and
at least a portion of each of the at least one piece of weight information and the at least one piece of bias information is associated with a second neural network, wherein the second neural network is associated with the at least a portion of each of the first model, the second model, the third model, and the fourth model.
14. The speech synthesis method of claim 12 , wherein the training on the parameter information is based on an output of the fourth neural network such that a correlation between the first set of speech data and the text data is learned.
15. A non-transitory computer-readable recording medium comprising a computer program for performing a speech synthesis method, the speech synthesis method comprising:
acquiring a second set of speech data and a target text;
acquiring, using a text-to-speech synthesis model trained based on text data corresponding to a first set of speech data and at least a portion of the first set of speech data, a first set of information, wherein the first set of information includes a first set of embedding information comprising the second set of speech data;
acquiring, using the text-to-speech synthesis model, a second set of information, wherein:
the second set of information includes a second set of embedding information, comprising embeddings of the second set of speech data,
the second set of embedding information is acquired by deploying an attention mechanism using query components generated based on a sequence of the target text, and
acquiring the second set of information comprises:
encoding the target text, and
extracting the query components from the encoded target text, wherein:
the query components are generated based on a sequence of the encoded target text, and
the sequence is generated based on the first set of information and the encoded target text;
acquiring audio data, using the text-to-speech synthesis model, wherein the audio data:
corresponds to the target text, and
reflects characteristics of speech of a speaker of the second set of speech data as a sound spectrum visualization generated based on the first set of information and the second set of information; and
deriving, using the text-to-speech synthesis model, a speech recording corresponding to the audio data.
16. The speech synthesis method of claim 9 , wherein:
the query components generated based on the sequence of the encoded target text make up an attention query,
K is an attention key, and
V is an attention value.