Method and device for speech synthesis based on multi-speaker training data sets
View Patent ↗An exemplary embodiment of the present disclosure is a speech synthesis method based on multi-speaker training dataset of a speech synthesis apparatus including pre-training a speech synthesis model using a previously stored neural network with a single speaker training dataset with most spoken sentences, among training datasets of a plurality of speakers, fine-tuning the pre-trained speech synthesis model with a training dataset of a plurality of speakers, and applying a target speech dataset to the fine-tuned speech synthesis model to be converted into a mel spectrogram.
1 . A speech synthesis method based on a multi-speaker training dataset of a speech synthesis apparatus, comprising:
pre-training by applying a single speaker training dataset with most spoken sentences, among training datasets of a plurality of speakers, to a speech synthesis model using a previously stored neural network;
fine-tuning the pre-trained speech synthesis model with a training dataset of the plurality of speakers; and
converting to a mel spectrogram by applying a target speech dataset to the fine-tuned speech synthesis model,
wherein the speech synthesis model conditions a target speaker based on a speaker embedding and a score vector, the score vector being calculated by:
calculating similarity scores between the target speaker and the plurality of speakers with respect to a speaking characteristic;
selecting a speaker, from among the plurality of speakers, based on the similarity scores, such that the selected speaker has the most similar speaking characteristic to the target speaker;
multiplying (i) a one-hot vector identifying the selected speaker from among the plurality of speakers, with (ii) a similarity score between the target speaker and the selected speaker; and
using a result of the multiplication as the score vector.
2 . The speech synthesis method based on a multi-speaker training dataset of claim 1 , further comprising:
generating a waveform type speech file based on the mel spectrogram.
3 . The speech synthesis method based on a multi-speaker training dataset of claim 1 , wherein the converting to a mel spectrogram further includes:
selecting a speaker with a high similarity to a target speech, among previously stored training datasets of the plurality of speakers, based on the target speech dataset; and
setting speaker embedding of the selected speaker as an initial value of the fine-tuned speech synthesis model.
4 . The speech synthesis method based on a multi-speaker training dataset of claim 3 , wherein in the selecting a speaker with a high similarity, the similarity is calculated based on a tone color and a speech rate between speakers.
5 . The speech synthesis method based on a multi-speaker training dataset of claim 4 , wherein the selecting a speaker with a high similarity includes:
calculating a similarity of a tone color by extracting a feature vector from the speech between the speakers and using the inner product;
calculating a phoneme duration of each speaker and calculating a similarity of a speech rate by dividing the phoneme duration of the trained speaker by a phoneme duration of the target speaker;
calculating a tone color similarity score between two speakers based on the similarity of the tone color and the similarity of the speech rate; and
selecting a speaker with the highest tone color similarity score as a speaker with the most similar speaking characteristic.
6 . The speech synthesis method based on a multi-speaker training dataset of claim 1 , wherein the speech synthesis model conditions each speaker based on a trainable speaker embedding and a one-hot vector.
7 . A speech synthesis apparatus based on a multi-speaker training dataset, comprising:
a memory which stores one or more instructions; and
a processor which executes the one or more instructions which are stored in the memory,
wherein the processor executes the one or more instructions to pre-train a speech synthesis model using a previously stored neural network with a single speaker training dataset with most spoken sentences, among training datasets of a plurality of speakers, fine-tune the pre-trained speech synthesis model with a training dataset of the plurality of speakers, and apply a target speech dataset to the fine-tuned speech synthesis model to be converted into a mel spectrogram, and
wherein the speech synthesis model conditions a target speaker based on a speaker embedding and a score vector, the score vector being calculated by:
calculating similarity scores between the target speaker and the plurality of speakers with respect to a speaking characteristic;
selecting a speaker, from among the plurality of speakers, based on the similarity scores, such that the selected speaker has the most similar speaking characteristic to the target speaker;
multiplying (i) a one-hot vector identifying the selected speaker from among the plurality of speakers, with (ii) a similarity score between the target speaker and the selected speaker; and
using a result of the multiplication as the score vector.
8 . The speech synthesis apparatus based on a multi-speaker training dataset of claim 7 , wherein the processor generates a waveform type speech file based on the mel spectrogram.
9 . The speech synthesis apparatus based on a multi-speaker training dataset of claim 7 , wherein the processor is configured to select a speaker with a high similarity to a target speech, among previously stored training datasets of the plurality of speakers, based on the target speech dataset; and set a speaker embedding of the selected speaker as an initial value of the fine-tuned speech synthesis model.
10 . The speech synthesis apparatus based on a multi-speaker training dataset of claim 9 , wherein the processor calculates the similarity based on the tone color and the speech rate between the speakers.
11 . The speech synthesis apparatus based on a multi-speaker training dataset of claim 10 , wherein the processor is configured to calculate a similarity of a tone color by extracting a feature vector from the speech between the speakers and using the inner product, calculate a phoneme duration of each of the speakers, calculate a similarity of a speech rate by dividing the phoneme duration of the trained speaker by a phoneme duration of the target speaker, calculate a tone color similarity score between two speakers based on the similarity of the tone color and the similarity of the speech rate, and select a speaker with the highest tone color similarity score as a speaker with the most similar speaking characteristic.
12 . The speech synthesis apparatus based on a multi-speaker training dataset of claim 7 , wherein the speech synthesis model conditions each speaker based on a trainable speaker embedding and a one-hot vector.