Granted Patent B2

US 12,469,482 · App. 18/042,077 · Granted Nov 11, 2025

Inventors: Chenchang Xu (Beijing, CN); Junjie Pan (Beijing, CN)

Assignee: BEIJING YOUZHUJU NETWORK TECHNOLOGY CO. LTD.

G10L13/04G10L13/08

Abstract

Claims (92)

1 . A method for speech synthesis, the method comprising:

inputting a plurality of training samples into a speech synthesis model for training the speech synthesis model, wherein the plurality of training samples comprise a first training sample, the first training sample comprising a first sample text with stress labels and a first sample audio of speaking the first sample text with a first tone, and wherein the plurality of training samples comprises a second training sample, the second training sample comprising a second sample text and a second sample audio of speaking the second sample text with a second tone;

generating a stress label vector based on vectorizing the stress labels corresponding to the first sample text and stress labels corresponding to the second sample text, wherein the stress labels corresponding to the second sample text are generated by the speech synthesis model;

generating a phoneme vector based on vectorizing a sequence of phonemes corresponding to the first sample text and the second sample text;

generating a tone label vector based on vectorizing tone labels, wherein the tone labels are extracted from the first sample audio and the second sample audio;

determining a Mel spectrum based on the phoneme vector, the stress label vector, and the tone label vector;

adjusting parameters of the speech synthesis model based at least in part on the Mel spectrum and at least one loss function to obtain a trained speech synthesis model;

acquiring a text to be synthesized with a target tone label, the target tone label indicating a target tone of a first speaker or a target tone of a second speaker; and

inputting the text to be synthesized into the trained speech synthesis model to generate audio corresponding to the text, wherein the generated audio has the target tone and stressed pronunciation.

2 . The method according to claim 1 , further comprising:

wherein the trained speech synthesis model generates the audio corresponding to the text to be synthesized by:

determining a sequence of phonemes corresponding to the text to be synthesized;

determining phoneme level stress labels according to the stress words; and

generating the audio corresponding to the text to be synthesized according to the sequence of phonemes, the stress labels and the target tone label.

3 . The method according to claim 2 , wherein the generating audio corresponding to the text to be synthesized according to the sequence of phonemes, the stress label and the target tone label comprises:

vectorizing the sequence of phonemes to obtain a phoneme vector;

vectorizing the stress labels to obtain a stress label vector;

vectorizing the target tone label to obtain a tone label vector;

determining a target phoneme vector according to the phoneme vector, the stress label vector and the tone label vector;

determining a Mel spectrum according to the target phoneme vector; and

inputting the Mel spectrum into a vocoder to obtain the audio corresponding to the text to be synthesized.

4 . The method according to claim 3 , wherein the determining the Mel spectrum according to the target phoneme vector comprises:

inputting the target phoneme vector into an encoder, and inputting a vector output by the encoder into a decoder to obtain corresponding Mel spectrum, wherein the encoder is used to determine pronunciation information of each phoneme in a sequence of phonemes corresponding to the input vector, and the decoder is used to perform conversion processing according to the pronunciation information of each phoneme corresponding to the input vector to obtain the Mel spectrum corresponding to each phoneme.

5 . The method according to claim 4 , wherein the target phoneme vector is obtained by splicing the phoneme vector, the stress label vector and the tone label vector.

6 . The method according to claim 3 , wherein the determining a target phoneme vector according to the phoneme vector, the stress label vector and the tone label vector comprises:

inputting the phoneme vector into an encoder, and determining the target phoneme vector according to the vector output by the encoder, the stress label vector and the tone label vector;

wherein the determining a Mel spectrum according to the target phoneme vector comprises:

inputting the target phoneme vector into a decoder to obtain the Mel spectrum;

wherein, the encoder is used to determine pronunciation information of each phoneme in a sequence of phonemes corresponding to the input vector, and the decoder is used to perform conversion processing according to the pronunciation information of each phoneme corresponding to the input vector to obtain the Mel spectrum corresponding to each phoneme.

7 . The method according to claim 6 , wherein the target phoneme vector is obtained by splicing the vector output by the encoder, the stress label vector and the tone label vector.

8 . The method according to claim 1 , wherein the at least one loss function comprises a first loss function, the method further comprising:

determining a target sample phoneme vector based on the phoneme vector, the stress label vector, and the tone label vector, wherein determining the Mel spectrum comprises determining the Mel spectrum based on the target sample phoneme vector; and

calculating the first loss function based on the Mel spectrum and the actual Mel spectrum of the sample audios corresponding to the target sample texts.

9 . The method according to claim 8 , wherein the plurality of first sample texts are a plurality of texts including different contents and the texts including the same content are marked initial stress labels by different users.

10 . The method according to claim 8 , wherein the process of training the speech synthesis model further comprises steps of:

generating a sequence of word vectors corresponding to the first sample text, and generating sample stress labels corresponding to the first sample text according to the stress words marked with the first sample text;

inputting the sequence of word vectors into the stress recognition module, and calculating a second loss function according to the result output by the stress recognition module and the sample stress label;

wherein the adjusting parameters of the speech synthesis model through the first loss function comprises:

adjusting parameters of the speech synthesis model through the first loss function and the second loss function.

11 . The method according to claim 10 , wherein the adjusting parameters of the speech synthesis model through the first loss function and the second loss function comprises:

performing weighted summation of the first loss function and the second loss function through weight values adaptively changed during the training process to obtain a target loss function;

adjusting parameters of the speech synthesis model according to the target loss function.

12 . The method according to claim 2 , wherein the stress words in the text to be synthesized are determined by steps of:

determining whether the text to be synthesized is marked with stress words;

in response to determine that the text to be synthesized is not marked with stress words, identifying stress words in the text to be synthesized through a stress recognition module; and

in response to determine that the text to be synthesized is marked with stress words, determining stress words in the text to be synthesized according to marking information corresponding to the stress words.

13 . A non-transitory computer-readable medium having computer programs stored thereon, which, when executed by a processing apparatus, execute operations comprising:

inputting a plurality of training samples into a speech synthesis model for training the speech synthesis model, wherein the plurality of training samples comprise a first training sample, the first training sample comprising a first sample text with stress labels and a first sample audio of speaking the first sample text with a first tone, and wherein the plurality of training samples comprises a second training sample, the second training sample comprising a second sample text and a second sample audio of speaking the second sample text with a second tone;

generating a stress label vector based on vectorizing the stress labels corresponding to the first sample text and stress labels corresponding to the second sample text, wherein the stress labels corresponding to the second sample text are generated by the speech synthesis model;

generating a phoneme vector based on vectorizing a sequence of phonemes corresponding to the first sample text and the second sample text;

generating a tone label vector based on vectorizing tone labels, wherein the tone labels are extracted from the first sample audio and the second sample audio;

determining a Mel spectrum based on the phoneme vector, the stress label vector, and the tone label vector;

adjusting parameters of the speech synthesis model based at least in part on the Mel spectrum and at least one loss function to obtain a trained speech synthesis model;

acquiring a text to be synthesized with a target tone label, the target tone label indicating a target tone of a first speaker or a target tone of a second speaker; and

inputting the text to be synthesized into the trained speech synthesis model to generate audio corresponding to the text, wherein the generated audio has the target tone and stressed pronunciation.

14 . An electronic device comprising:

a storage apparatus having a computer program stored thereon;

a processing apparatus configured to execute the computer program in the storage apparatus, so as to execute operations comprising:

inputting a plurality of training samples into a speech synthesis model for training the speech synthesis model, wherein the plurality of training samples comprise a first training sample, the first training sample comprising a first sample text with stress labels and a first sample audio of speaking the first sample text with a first tone, and wherein the plurality of training samples comprises a second training sample, the second training sample comprising a second sample text and a second sample audio of speaking the second sample text with a second tone;

generating a stress label vector based on vectorizing the stress labels corresponding to the first sample text and stress labels corresponding to the second sample text, wherein the stress labels corresponding to the second sample text are generated by the speech synthesis model;

generating a phoneme vector based on vectorizing a sequence of phonemes corresponding to the first sample text and the second sample text;

generating a tone label vector based on vectorizing tone labels, wherein the tone labels are extracted from the first sample audio and the second sample audio;

determining a Mel spectrum based on the phoneme vector, the stress label vector, and the tone label vector;

adjusting parameters of the speech synthesis model based at least in part on the Mel spectrum and at least one loss function to obtain a trained speech synthesis model;

acquiring a text to be synthesized with a target tone label, the target tone label indicating a target tone of a first speaker or a target tone of a second speaker; and

inputting the text to be synthesized into the trained speech synthesis model to generate audio corresponding to the text, wherein the generated audio has the target tone and stressed pronunciation.

15 . The electronic device according to claim 14 , the operations further comprising:

wherein the trained speech synthesis model is configured to generate the audio corresponding to the text to be synthesized by:

determining a sequence of phonemes corresponding to the text to be synthesized;

determining phoneme level stress labels according to the stress words; and

generating the audio corresponding to the text to be synthesized according to the sequence of phonemes, the stress labels and the target tone label.

16 . The electronic device according to claim 15 , wherein the generating the audio corresponding to the text to be synthesized according to the sequence of phonemes, the stress label and the target tone label comprises:

vectorizing the sequence of phonemes to obtain a phoneme vector;

vectorizing the stress labels to obtain a stress label vector;

vectorizing the target tone label to obtain a tone label vector;

determining a target phoneme vector according to the phoneme vector, the stress label vector and the tone label vector;

determining a Mel spectrum according to the target phoneme vector; and

inputting the Mel spectrum into a vocoder to obtain the audio corresponding to the text to be synthesized.

17 . The electronic device according to claim 16 , wherein the determining the Mel spectrum according to the target phoneme vector comprises:

inputting the target phoneme vector into an encoder, and inputting a vector output by the encoder into a decoder to obtain corresponding Mel spectrum, wherein the encoder is used to determine pronunciation information of each phoneme in a sequence of phonemes corresponding to the input vector, and the decoder is used to perform conversion processing according to the pronunciation information of each phoneme corresponding to the input vector to obtain the Mel spectrum corresponding to each phoneme.

18 . The electronic device according to claim 16 , wherein the determining a target phoneme vector according to the phoneme vector, the stress label vector and the tone label vector comprises:

inputting the phoneme vector into an encoder, and determining the target phoneme vector according to the vector output by the encoder, the stress label vector and the tone label vector;

wherein the determining a Mel spectrum according to the target phoneme vector comprises:

inputting the target phoneme vector into a decoder to obtain the Mel spectrum;

wherein, the encoder is used to determine pronunciation information of each phoneme in a sequence of phonemes corresponding to the input vector, and the decoder is used to perform conversion processing according to the pronunciation information of each phoneme corresponding to the input vector to obtain the Mel spectrum corresponding to each phoneme.

19 . The electronic device according to claim 14 , wherein the at least one loss function comprises a first loss function, the operations further comprising:

determining a target sample phoneme vector based on the phoneme vector, the stress label vector and the tone label vector, wherein determining the Mel spectrum comprises determining the Mel Spectrum based on to the target sample phoneme vector; and

calculating the first loss function based on the Mel spectrum and the actual Mel spectrum of the sample audios corresponding to the target sample texts.

20 . The electronic device according to claim 15 , wherein the stress words in the text to be synthesized are determined by steps of:

determining whether the text to be synthesized is marked with stress words;

in response to determine that the text to be synthesized is not marked with stress words, identifying stress words in the text to be synthesized through a stress recognition module; and

in response to determine that the text to be synthesized is marked with stress words, determining stress words in the text to be synthesized according to marking information corresponding to the stress words.

Assignments (2)

Priority Claims (1)

Continuity (1)

References Cited (29)