Electronic device and method of generating text-to-speech model for prosody control of the electronic device
According to certain embodiments, an electronic device, comprises: a memory storing therein instructions; and a processor electrically connected to the memory and configured to execute the instructions, wherein, when the instructions are executed by the processor, the processor receives training data comprising a plurality of phenomes; determines a prosody value for each one of the plurality of phenomes in the training data; clusters the plurality of phenomes based on the prosody value for each one of the plurality of phenomes in the training data, thereby resulting in a plurality of prosody clusters; extracts a phoneme sequence corresponding to a text in the training data; extracts a prosody cluster index sequence corresponding to an utterance of the text by selecting one of the plurality of clusters based on prosody values of the utterance of the text; and generates a text-to-speech (TTS) model based on the phoneme sequence and the prosody cluster index sequence.
1 . An electronic device, comprising:
a memory storing therein instructions; and
a processor electrically connected to the memory and configured to execute the instructions,
wherein, when the instructions are executed by the processor, the processor is configured to:
receive training data comprising a plurality of phonemes;
calculate a prosody value for each one of the plurality of phonemes in the training data;
cluster the plurality of phonemes based on the prosody value for each one of the plurality of phonemes in the training data, thereby resulting in a plurality of prosody clusters;
extract a phoneme sequence corresponding to a text in the training data;
extract a prosody cluster index sequence corresponding to an utterance of the text by selecting one of the plurality of prosody clusters based on prosody values of the utterance of the text; and
generate a text-to-speech (TTS) model based on the phoneme sequence and the prosody cluster index sequence, wherein the TTS model comprises a phoneme model, a prosody model that is trained separately from the phoneme model, and a decoding model,
wherein the generating the TTS model comprises:
predicting, via the phoneme model, a first length information of a spectrogram frame corresponding to each phoneme of the phoneme sequence based on phoneme characteristics independently from the prosody model, the phoneme characteristics are extracted from the phoneme sequence;
predicting, via the prosody model, a second length information of a spectrogram frame corresponding to each phoneme of the phoneme sequence based on prosody characteristics independently from the phoneme model, the prosody characteristics are extracted from the prosody cluster index sequence;
generating, via the phoneme model, length-corrected phoneme characteristics based on the first length information of the spectrogram frame corresponding to each phoneme of the phoneme sequence;
generating, via the prosody model, length-corrected prosody characteristics based on the second length information of the spectrogram frame corresponding to each phoneme of the phoneme sequence;
generating a spectrogram corresponding to the utterance of the text based on inputting the length-corrected phoneme characteristics and the length-corrected prosody characteristics; and
training the phoneme model, the prosody model, and the decoding model based on an error value between the spectrogram corresponding to the utterance of the text and a ground truth spectrogram.
2 . The electronic device of claim 1 , wherein the processor is configured to:
when the prosody cluster index sequence comprises a prosody cluster index sequence extracted for each prosody, train each prosody model corresponding to each prosody using the prosody cluster index sequence extracted for each prosody, and
wherein each cluster of the plurality of prosody clusters includes different phonemes having a matching prosody value.
3 . The electronic device of claim 1 , wherein each of the plurality of prosody clusters represents a prosody degree.
4 . The electronic device of claim 1 , wherein the prosody values of all of the plurality of phonemes comprise prosody values of the plurality of phonemes extracted for each prosody.
5 . The electronic device of claim 1 , wherein the processor is configured to:
determine the prosody clusters from a distribution of the prosody values of the plurality of phonemes by performing the clustering on all the phonemes.
6 . The electronic device of claim 1 , wherein, the processor is configured to:
cluster the plurality of phonemes differently based on a prosody characteristic.
7 . The electronic device of claim 6 , wherein the processor is configured to:
perform the clustering on values of first prosody among the prosody values of all the phonemes, regardless of the phonemes; and
perform the clustering on values of second prosody among the prosody values of all the phonemes by classifying the values of the second prosody by each phoneme.
8 . The electronic device of claim 7 , wherein the first prosody comprises a pitch, and the second prosody comprises an utterance length.
9 . An operation method of an electronic device, comprising:
extracting a phoneme sequence corresponding to a text;
extracting a prosody cluster index sequence corresponding to an utterance of the text by matching prosody values of the utterance to at least one of a plurality of prosody clusters, wherein each of the plurality of prosody clusters representing a prosody degree, wherein the plurality of prosody clusters are clustered based on a calculated prosody for a plurality of phonemes; and
generating a text-to-speech (TTS) model based on the phoneme sequence and the prosody cluster index sequence,
wherein the TTS model comprises a phoneme model, a prosody model that is trained separately from the phoneme model, and a decoding model,
wherein the generating the TTS model comprises:
predicting, via the phoneme model, a first length information of a spectrogram frame corresponding to each phoneme of the phoneme sequence based on phoneme characteristics independently from the prosody model, the phoneme characteristics are extracted from the phoneme sequence;
predicting, via the prosody model, a second length information of a spectrogram frame corresponding to each phoneme of the phoneme sequence based on prosody characteristics independently from the phoneme model, the prosody characteristics are extracted from the prosody cluster index sequence;
generating, via the phoneme model, length-corrected phoneme characteristics based on the first length information of the spectrogram frame corresponding to each phoneme of the phoneme sequence;
generating, via the prosody model, length-corrected prosody characteristics based on the second length information of the spectrogram frame corresponding to each phoneme of the phoneme sequence;
generating a spectrogram corresponding to the utterance of the text based on inputting the length-corrected phoneme characteristics and the length-corrected prosody characteristics; and
training the phoneme model, the prosody model, and the decoding model based on an error value between the spectrogram corresponding to the utterance of the text and a ground truth spectrogram.
10 . The operation method of claim 9 , wherein the training of the prosody model comprises:
when the prosody cluster index sequence comprises a prosody cluster index sequence extracted for each prosody, training each prosody model corresponding to each prosody using the prosody cluster index sequence extracted for each prosody, and
wherein each cluster of the plurality of prosody clusters includes different phonemes having a matching prosody value.
11 . The operation method of claim 9 , wherein prosody values of all phonemes comprise prosody values of all the phonemes extracted for each prosody.
12 . The operation method of claim 9 , further comprising:
determining the prosody clusters by performing clustering on all phonemes in training data based on the prosody values of all the phonemes in the training data.
13 . The operation method of claim 12 , wherein the determining of the prosody clusters comprises:
performing the clustering on all the phonemes differently based on a prosody characteristic.
14 . The operation method of claim 13 , wherein the performing of the clustering differently comprises:
clustering values of first prosody among the prosody values of all the phonemes, regardless of the phonemes; and
clustering values of second prosody among the prosody values of all the phonemes by classifying the values by each phoneme.
15 . The operation method of claim 14 , wherein the first prosody comprises a pitch, and the second prosody comprises an utterance length.
16 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the operation method of claim 9 .