Systems and methods for text-to-speech synthesis using spoken example
View Patent ↗Systems and methods for speech synthesis and, in particular, text-to-speech systems and methods for converting a text input to a synthetic waveform by processing prosodic and phonetic content of a spoken example of the text input to accurately mimic the input speech style and pronunciation. Systems and methods provide an interface to a TTS system to allow a user to input a text string and a spoken utterance of the text string, extract prosodic parameters from the spoken input, and process the prosodic parameters to derive corresponding markup for the text input to enable a more natural sounding synthesized speech.
1. An article of manufacture comprising a program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform a method for speech synthesis that allows user specified pronunciations, the method comprising:
providing a user interface that allows a user to identify a text string for synthesis and to speak a pronunciation of the text string;
recording the user's spoken pronunciation of the text string as an audio signal;
extracting prosodic parameter values from the audio signal corresponding to the user's pronunciation of the text string, wherein extracting the prosodic parameter values comprises extracting duration parameter values from the audio signal by aligning the audio signal with the text string;
automatically translating at least a portion of the prosodic parameter values extracted at least in part by aligning the audio signal of the user's spoken pronunciation with the text string into abstract labels to generate a high-level markup of the text string; and
generating a synthetic speech waveform by applying a markup-enabled text-to-speech engine to the text string with the high-level markup.
2. The article of manufacture of claim 1 , wherein the extracting duration parameter values by aligning comprises segmenting the audio signal into time-segmented regions, wherein each time-segmented region is mapped to a corresponding phoneme.
3. The article of manufacture of claim 1 , wherein the extracting duration parameter values by aligning comprises using a Viterbi alignment process.
4. The article of manufacture of claim 1 , wherein the method further comprises directly specifying at least one portion of the prosodic parameter values as attribute values for mark-up elements.
5. The article of manufacture of claim 1 , wherein the translating comprises generating the markup of the text string using SSML (speech synthesis markup language).
6. The article of manufacture of claim 1 , further comprising instructions for processing phonetic content of the audio signal to generate the synthetic speech waveform having a desired pronunciation.
7. The article of manufacture of claim 1 , wherein the method further comprises extracting acoustic feature data from the audio signal and wherein the aligning further comprises outputting one or more duration contours.
8. The article of manufacture of claim 7 , wherein extracting acoustic feature data from the audio signal comprises digitizing the audio signal into a set of frames and transforming the digitized audio signal into a set of feature vectors on a frame-by-frame basis.
9. The article of manufacture of claim 1 , wherein the method further comprises directly specifying at least one portion of the extracted prosodic parameter values as prosodic parameter values for synthesis of the synthetic speech waveform representing the text string.
10. A text-to-speech (TTS) system that allows user specified pronunciations, the system comprising:
at least one processor; and
at least one storage device storing processor-executable instructions that, when executed by the at least one processor, perform a method comprising:
providing a user interface that allows a user to identify a text string for synthesis and to speak a pronunciation of the text string;
recording the user's spoken pronunciation of the text string as an audio signal;
extracting prosodic parameter values from the audio signal corresponding to the user's pronunciation of the text string, wherein extracting the prosodic parameter values comprises extracting duration parameter values from the audio signal by aligning the audio signal with the text string;
automatically translating at least a portion of the prosodic parameter values extracted at least in part by aligning the audio signal of the user's spoken pronunciation with the text string into abstract labels to generate a high-level markup of the text string; and
generating a synthetic speech waveform by applying a markup-enabled text-to-speech engine to the text string with the high-level markup.
11. The article of manufacture of claim 8 , wherein transforming the digitized audio signal comprises producing a 24-dimensional cepstra feature vector for every 10 ms of the audio signal, concatenating frames to the left and to the right of a current frame to augment a current cepstral vector, and reducing each augmented cepstral vector to a 60-dimensional feature vector using linear discriminant analysis.
12. The system of claim 10 , wherein the method further comprises extracting acoustic feature data from the audio signal, and wherein the aligning comprises outputting one or more duration contours.
13. A method for speech synthesis that allows user specified pronunciations, the method comprising:
providing a user interface that allows a user to identify a text string for synthesis and to speak a pronunciation of the text string;
recording the user's spoken pronunciation of the text string as an audio signal;
extracting prosodic parameter values from the audio signal corresponding to the user's pronunciation of the text string, wherein extracting the prosodic parameter values comprises extracting duration parameter values from the audio signal by aligning the audio signal with the text string;
automatically translating at least a portion of the prosodic parameter values extracted at least in part by aligning the audio signal of the user's spoken pronunciation with the text string into abstract labels to generate a high-level markup of the text string; and
generating a synthetic speech waveform by applying a markup-enabled text-to-speech engine to the text string with the high-level markup.
14. The method of claim 13 , wherein the aligning comprises extracting acoustic feature data from the audio signal and time-aligning the audio signal to the text string using the acoustic feature data.
15. The method of claim 13 , wherein the aligning is performed using a Viterbi alignment process.
16. The method of claim 13 , further comprising directly specifying at least one portion of the prosodic parameter values as attribute values for mark-up elements.
17. The method of claim 13 , wherein the translating comprises generating the markup of the text string using SSML (speech synthesis markup language).
18. The method of claim 13 , further comprising processing phonetic content of the audio signal to generate the synthetic speech waveform having a desired pronunciation.
19. The method of claim 13 , wherein the aligning further comprises outputting one or more duration contours.
20. The method of claim 13 , further comprising extracting acoustic feature data from the audio signal, including digitizing the audio signal into a set of frames and transforming the digitized audio signal into a set of feature vectors on a frame-by-frame basis.
21. The method of claim 20 , wherein transforming the digitized audio signal comprises producing a 24-dimensional cepstra feature vector for every 10 ms of the audio signal, concatenating frames to the left and to the right of a current frame to augment a current cepstral vector, and reducing each augmented cepstral vector to a 60-dimensional feature vector using linear discriminant analysis.
22. The method of claim 13 , further comprising directly specifying at least one portion of the extracted prosodic parameter values as prosodic parameter values for synthesis of the synthetic speech waveform representing the text string.