Speech synthesis method and apparatus, electronic device, and readable storage medium

Inventors: Dongyang Dai (Beijing, CN); Yuanzhe Chen (Beijing, CN); Li Chen (Beijing, CN); Yuping Wang (Beijing, CN); Qiao Tian (Beijing, CN); Ming Tu (Los Angeles, CA); Rui Xia (Los Angeles, CA); Yuxuan Wang (Los Angeles, CA)

Assignee: Lemon Inc.

G10L13/027 G10L25/18

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12676138
App. No.: 18/568,261
Granted: Jul 7, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Provided are an audio synthesis method and apparatus, an electronic device, and a readable storage medium. In the present solution, conversion from a text to an audio having a target timbre is achieved by means of a pre-trained voice synthesis model, the voice synthesis model comprising a first feature extraction sub-model and a second feature extraction sub-model, wherein the first feature extraction sub-model outputs, according to an inputted text to be processed, an acoustic feature comprising a bottleneck feature; the second feature extraction sub-model outputs, according to the inputted first acoustic features, a Mel spectrum feature corresponding to the text to be processed; according to the Mel spectrum feature corresponding to the text to be processed, the target audio corresponding to the text to be processed is obtained, and the target audio has the target timbre.

Claims (41)

1 . A method for speech synthesis, comprising:

obtaining text to be synthesized into target audio having a target timbre;

generating a first acoustic feature corresponding to the text based on inputting the text into a first sub-model of a target speech synthesis model, wherein the first acoustic feature comprises a bottleneck feature that does not include information related to the target timbre;

generating a Mel spectrum sequence corresponding to the text based on inputting the first acoustic feature into a second sub-model of the target speech synthesis model, wherein the Mel spectrum sequence includes information related to the target timbre; and

synthesizing the target audio having the target timbre based on the Mel spectrum sequence.

2 . The method of claim 1 , wherein the first sub-model is obtained by training based on labeled text corresponding to a first sample audio and a second acoustic feature corresponding to the first sample audio, the second acoustic feature including a first labeled bottleneck feature corresponding to the first sample audio.

3 . The method of claim 2 , wherein the second sub-model is obtained by training based on a third acoustic feature and a first labeled Mel spectrum feature corresponding to a second sample audio, and a fourth acoustic feature and a second labeled Mel spectrum feature corresponding to a third sample audio;

wherein the third acoustic feature includes a second labeled bottleneck feature corresponding to the second sample audio; the fourth acoustic feature includes a third labeled bottleneck feature corresponding to the third sample audio; and the third sample audio is a sample audio having the target timbre.

4 . The method of claim 3 , wherein the first labeled bottleneck feature corresponding to the first sample audio, the second labeled bottleneck feature corresponding to the second sample audio and the third labeled bottleneck feature corresponding to the third sample audio are obtained by performing, using an encoder of an end-to-end speech recognition model, bottleneck feature extraction on the first sample audio, the second sample audio and the third sample audio as input respectively.

5 . The method of claim 3 , wherein the second acoustic feature further includes a first labeled baseband feature corresponding to the first sample audio; the third acoustic feature further includes a second labeled baseband feature corresponding to the second sample audio; and the fourth acoustic feature further includes a third labeled baseband feature corresponding to the third sample audio; and

correspondingly, the first acoustic feature output by the first sub-model further includes a baseband feature corresponding to the text to be processed.

6 . The method of claim 5 , wherein the first labeled baseband feature corresponding to the first sample audio, the second labeled baseband feature corresponding to the second sample audio and the third labeled baseband feature corresponding to the third sample audio are obtained by performing digital signal processing on the first sample audio, the second sample audio and the third audio respectively.

7 . The method of claim 3 , wherein a language of the first sample audio is the same as that of the second sample audio; and the language of the first sample audio is different from that of the third sample audio.

8 . An electronic device, comprising:

a memory having computer program instructions stored thereon, and

a processor, wherein computer program instructions, when executed by the processor, cause the electronic device to perform operations comprising:

obtaining text to be synthesized into target audio having a target timbre;

synthesizing the target audio having the target timbre based on the Mel spectrum sequence.

9 . The electronic device of claim 8 , wherein the first sub-model is obtained by training based on labeled text corresponding to a first sample audio and a second acoustic feature corresponding to the first sample audio, the second acoustic feature including a first labeled bottleneck feature corresponding to the first sample audio.

10 . The electronic device of claim 9 , wherein the second sub-model is obtained by training based on a third acoustic feature and a first labeled Mel spectrum feature corresponding to a second sample audio, and a fourth acoustic feature and a second labeled Mel spectrum feature corresponding to a third sample audio;

11 . The electronic device of claim 10 , wherein the first labeled bottleneck feature corresponding to the first sample audio, the second labeled bottleneck feature corresponding to the second sample audio and the third labeled bottleneck feature corresponding to the third sample audio are obtained by performing, using an encoder of an end-to-end speech recognition model, bottleneck feature extraction on the first sample audio, the second sample audio and the third sample audio as input respectively.

12 . The electronic device of claim 10 , wherein the second acoustic feature further includes a first labeled baseband feature corresponding to the first sample audio;

the third acoustic feature further includes a second labeled baseband feature corresponding to the second sample audio; and the fourth acoustic feature further includes a third labeled baseband feature corresponding to the third sample audio; and

correspondingly, the first acoustic feature output by the first sub-model further includes a baseband feature corresponding to the text to be processed.

13 . The electronic device of claim 12 , wherein the first labeled baseband feature corresponding to the first sample audio, the second labeled baseband feature corresponding to the second sample audio and the third labeled baseband feature corresponding to the third sample audio are obtained by performing digital signal processing on the first sample audio, the second sample audio and the third audio respectively.

14 . The electronic device of claim 10 , wherein a language of the first sample audio is the same as that of the second sample audio; and the language of the first sample audio is different from that of the third sample audio.

15 . A non-transitory readable storage medium, comprising computer program instructions which, when executed by at least one processor of an electronic device, cause the electronic device to perform operations comprising:

obtaining text to be synthesized into target audio having a target timbre;

synthesizing the target audio having the target timbre based on the Mel spectrum sequence.

16 . The non-transitory readable storage medium of claim 15 , wherein the first sub-model is obtained by training based on labeled text corresponding to a first sample audio and a second acoustic feature corresponding to the first sample audio, the second acoustic feature including a first labeled bottleneck feature corresponding to the first sample audio.

17 . The non-transitory readable storage medium of claim 16 , wherein the second sub-model is obtained by training based on a third acoustic feature and a first labeled Mel spectrum feature corresponding to a second sample audio, and a fourth acoustic feature and a second labeled Mel spectrum feature corresponding to a third sample audio;

18 . The non-transitory readable storage medium of claim 17 , wherein the first labeled bottleneck feature corresponding to the first sample audio, the second labeled bottleneck feature corresponding to the second sample audio and the third labeled bottleneck feature corresponding to the third sample audio are obtained by performing, using an encoder of an end-to-end speech recognition model, bottleneck feature extraction on the first sample audio, the second sample audio and the third sample audio as input respectively.

19 . The non-transitory readable storage medium of claim 17 , wherein the second acoustic feature further includes a first labeled baseband feature corresponding to the first sample audio; the third acoustic feature further includes a second labeled baseband feature corresponding to the second sample audio; and the fourth acoustic feature further includes a third labeled baseband feature corresponding to the third sample audio; and

correspondingly, the first acoustic feature output by the first sub-model includes a baseband feature corresponding to the text to be processed.

20 . The non-transitory readable storage medium of claim 19 , wherein the first labeled baseband feature corresponding to the first sample audio, the second labeled baseband feature corresponding to the second sample audio and the third labeled baseband feature corresponding to the third sample audio are obtained by performing digital signal processing on the first sample audio, the second sample audio and the third audio respectively.