Method, apparatus, storage medium, and electronic device for speech synthesis

1 . A method for speech synthesis, the method comprising:

acquiring a text to be synthesized marked with stress words;

inputting the text to be synthesized into a speech synthesis model to obtain audio information corresponding to the text to be synthesized, the speech synthesis model being obtained by training on sample texts marked with stress words and sample audios corresponding to the sample texts, wherein the stress words marked in the sample texts are determined by:

acquiring a plurality of sample texts each of which includes stress words with initial stress labels,

for each of the stress words marked with the initial stress label, if the stress word is marked as a stress word in each of the sample texts, adding a target stress label to the stress word; if the stress word is marked as a stress word in at least two of the sample texts, in a case that the fundamental frequency of the stress word is greater than a preset fundamental frequency threshold and the energy of the stress word is greater than a preset energy threshold, adding a target stress label to the stress word,

determining stress words to each of which the target stress label has been added as the stress words marked in the sample texts; and

generating, by the speech synthesis model, the audio information corresponding to the text to be synthesized based on a sequence of phonemes corresponding to the text and stress labels corresponding to the stress words marked in the text, wherein the generating audio information corresponding to the text to be synthesized based on the sequence of phonemes and the stress labels comprises:

vectorizing the sequence of phonemes corresponding to the text to be synthesized to obtain a phoneme vector,

vectorizing the stress labels to obtain a stress label vector,

determining a target phoneme vector according to the phoneme vector and the stress label vector,

determining a Mel spectrum according to the target phoneme vector, and

inputting the Mel spectrum into a vocoder to obtain the audio information corresponding to the text to be synthesized.

2 . The method according to claim 1 , wherein the speech synthesis model is used to process the text to be synthesized by:

determining the sequence of phonemes corresponding to the text to be synthesized; and

determining the stress labels corresponding to the stress words marked in the text to be synthesized.

3 . The method according to claim 1 , wherein the determining a Mel spectrum according to the target phoneme vector comprises:

inputting the target phoneme vector into an encoder, and inputting a vector output by the encoder into a decoder to obtain corresponding Mel spectrum, wherein the encoder is used to determine pronunciation information of each phoneme in a sequence of phonemes corresponding to the input vector, and the decoder is used to perform conversion processing according to the pronunciation information of each phoneme corresponding to the input vector to obtain the Mel spectrum corresponding to each phoneme.

4 . The method according to claim 3 , wherein the target phoneme vector is obtained by splicing the phoneme vector and the stress label vector.

5 . The method according to claim 1 , wherein the determining a target phoneme vector according to the phoneme vector and the stress label vector comprises:

inputting the phoneme vector into an encoder, and determining the target phoneme vector according to the vector output by the encoder and the stress label vector;

the determining a Mel spectrum according to the target phoneme vector comprises:

inputting the target phoneme vector into a decoder to obtain the Mel spectrum;

wherein, the encoder is used to determine pronunciation information of each phoneme in a sequence of phonemes corresponding to the input vector, and the decoder is used to perform conversion processing according to the pronunciation information of each phoneme corresponding to the input vector to obtain the Mel spectrum corresponding to each phoneme.

6 . The method according to claim 5 , wherein the target phoneme vector is obtained by splicing the vector output by the encoder and the stress label vector.

7 . The method according to claim 1 , wherein the plurality of sample texts are a plurality of texts including different contents and the texts including the same content are marked initial stress labels by different users.

8 . The method according to claim 1 , wherein the initial stress labels in the sample text correspond to prosodic phrases.

9 . The method according to claim 1 , wherein the speech synthesis model is obtained by training in the following manner:

vectorizing a sequence of phonemes corresponding to the sample text to obtain a sample phoneme vector;

determining sample stress labels corresponding to the sample text according to the stress words marked in the sample text, and vectorizing the sample stress labels to obtain a phoneme level sample stress label vector;

determining a target sample phoneme vector according to the sample phoneme vector and the sample stress label vector, and determining a sample Mel spectrum according to the target sample phoneme vector;

calculating a loss function according to the sample Mel spectrum and the actual Mel spectrum corresponding to the sample audio, and adjusting parameters of the speech synthesis model through the loss function.

10 . A non-transitory computer-readable medium having a computer program stored thereon, the program, when executed by a processing apparatus, execute operations comprising:

acquiring a text to be synthesized marked with stress words;

inputting the text to be synthesized into a speech synthesis model to obtain audio information corresponding to the text to be synthesized, the speech synthesis model being obtained by training on sample texts marked with stress words and sample audios corresponding to the sample texts, wherein the stress words marked in the sample texts are determined by:

acquiring a plurality of sample texts each of which includes stress words with initial stress labels,

for each of the stress words marked with the initial stress label, if the stress word is marked as a stress word in each of the sample texts, adding a target stress label to the stress word; if the stress word is marked as a stress word in at least two of the sample texts, in a case that the fundamental frequency of the stress word is greater than a preset fundamental frequency threshold and the energy of the stress word is greater than a preset energy threshold, adding a target stress label to the stress word,

determining stress words to each of which the target stress label has been added as the stress words marked in the sample texts; and

generating, by the speech synthesis model, the audio information corresponding to the text to be synthesized based on a sequence of phonemes corresponding to the text and stress labels corresponding to the stress words marked in the text, wherein the generating audio information corresponding to the text to be synthesized based on the sequence of phonemes and the stress labels comprises:

vectorizing the sequence of phonemes corresponding to the text to be synthesized to obtain a phoneme vector,

vectorizing the stress labels to obtain a stress label vector,

determining a target phoneme vector according to the phoneme vector and the stress label vector,

determining a Mel spectrum according to the target phoneme vector, and

inputting the Mel spectrum into a vocoder to obtain the audio information corresponding to the text to be synthesized.

11 . An electronic device, comprising:

a storage apparatus having computer programs stored thereon;

a processing apparatus configured to execute the computer programs in the storage apparatus, so as to execute operations comprising:

acquiring a text to be synthesized marked with stress words;

inputting the text to be synthesized into a speech synthesis model to obtain audio information corresponding to the text to be synthesized, the speech synthesis model being obtained by training on sample texts marked with stress words and sample audios corresponding to the sample texts, wherein the stress words marked in the sample texts are determined by:

acquiring a plurality of sample texts each of which includes stress words with initial stress labels,

for each of the stress words marked with the initial stress label, if the stress word is marked as a stress word in each of the sample texts, adding a target stress label to the stress word; if the stress word is marked as a stress word in at least two of the sample texts, in a case that the fundamental frequency of the stress word is greater than a preset fundamental frequency threshold and the energy of the stress word is greater than a preset energy threshold, adding a target stress label to the stress word,

determining stress words to each of which the target stress label has been added as the stress words marked in the sample texts; and

generating, by the speech synthesis model, the audio information corresponding to the text to be synthesized based on a sequence of phonemes corresponding to the text and stress labels corresponding to the stress words marked in the text, wherein the generating audio information corresponding to the text to be synthesized based on the sequence of phonemes and the stress labels comprises:

vectorizing the sequence of phonemes corresponding to the text to be synthesized to obtain a phoneme vector,

vectorizing the stress labels to obtain a stress label vector,

determining a target phoneme vector according to the phoneme vector and the stress label vector,

determining a Mel spectrum according to the target phoneme vector, and

inputting the Mel spectrum into a vocoder to obtain the audio information corresponding to the text to be synthesized.

12 . The electronic device according to claim 11 , wherein the speech synthesis model is used to process the text to be synthesized by:

determining the sequence of phonemes corresponding to the text to be synthesized;

determining the stress labels corresponding to the stress words marked in the text to be synthesized.

13 . The electronic device according to claim 11 , wherein the determining a Mel spectrum according to the target phoneme vector comprises:

inputting the target phoneme vector into an encoder, and inputting a vector output by the encoder into a decoder to obtain corresponding Mel spectrum, wherein the encoder is used to determine pronunciation information of each phoneme in a sequence of phonemes corresponding to the input vector, and the decoder is used to perform conversion processing according to the pronunciation information of each phoneme corresponding to the input vector to obtain the Mel spectrum corresponding to each phoneme.

14 . The electronic device according to claim 13 , wherein the target phoneme vector is obtained by splicing the phoneme vector and the stress label vector.

15 . The electronic device according to claim 11 , wherein the determining a target phoneme vector according to the phoneme vector and the stress label vector comprises:

inputting the phoneme vector into an encoder, and determining the target phoneme vector according to the vector output by the encoder and the stress label vector;

the determining a Mel spectrum according to the target phoneme vector comprises:

inputting the target phoneme vector into a decoder to obtain the Mel spectrum;

wherein, the encoder is used to determine pronunciation information of each phoneme in a sequence of phonemes corresponding to the input vector, and the decoder is used to perform conversion processing according to the pronunciation information of each phoneme corresponding to the input vector to obtain the Mel spectrum corresponding to each phoneme.

16 . The electronic device according to claim 11 , wherein the speech synthesis model is obtained by training in the following manner:

vectorizing a sequence of phonemes corresponding to the sample text to obtain a sample phoneme vector;

determining sample stress labels corresponding to the sample text according to the stress words marked in the sample text, and vectorizing the sample stress labels to obtain a phoneme level sample stress label vector;

determining a target sample phoneme vector according to the sample phoneme vector and the sample stress label vector, and determining a sample Mel spectrum according to the target sample phoneme vector;

calculating a loss function according to the sample Mel spectrum and the actual Mel spectrum corresponding to the sample audio, and adjusting parameters of the speech synthesis model through the loss function.

17 . The non-transitory computer-readable medium according to claim 10 , wherein the speech synthesis model processes the text to be synthesized by:

determining the sequence of phonemes corresponding to the text to be synthesized; and

determining the stress labels corresponding to the stress words marked in the text to be synthesized.

18 . The non-transitory computer-readable medium according to claim 10 , wherein the determining a Mel spectrum according to the target phoneme vector comprises:

inputting the target phoneme vector into an encoder, and inputting a vector output by the encoder into a decoder to obtain corresponding Mel spectrum, wherein the encoder is configured to determine pronunciation information of each phoneme in a sequence of phonemes corresponding to the input vector, and the decoder is configured to perform conversion processing according to the pronunciation information of each phoneme corresponding to the input vector to obtain the Mel spectrum corresponding to each phoneme.