System and method for predicting prosodic parameters
View Patent ↗A method for generating a prosody model that predicts prosodic parameters is disclosed. Upon receiving text annotated with acoustic features, the method comprises generating first classification and regression trees (CARTs) that predict durations and F0 from text by generating initial boundary labels by considering pauses, generating initial accent labels by applying a simple rule on text-derived features only, adding the predicted accent and boundary labels to feature vectors, and using the feature vectors to generate the first CARTs. The first CARTs are used to predict accent and boundary labels. Next, the first CARTs are used to generate second CARTs that predict durations and F0 from text and acoustic features by using lengthened accented syllables and phrase-final syllables, refining accent and boundary models simultaneously, comparing actual and predicted duration of a whole prosodic phrase to normalize speaking rate, and generating the second CARTs that predict the normalized speaking rate.
1. A computing that predicts prosodic parameters from annotated speech files having annotations, the computing device comprising:
a processor;
a first module that, via the processor, generates initial accent and boundary labels from annotated speech files based on binary decisions; and
a second module that uses the generated initial accent and boundary labels to iteratively grow a classification and regression tree to generate improved classification and regression trees for predicting prosody parameters from text by:
(1) adding predicted linguistic features to text-derived annotations in the speech files;
(2) adding normalized syllable durations to the annotations;
(3) adding a plurality of extracted acoustic features to the annotations;
(4) generating initial accent and boundary labels by considering pauses and relative syllable durations;
(5) training classification and regression trees to predict durations and F0s from the predicted linguistic features and the internal accent and boundary labels;
(6) training refined classification and regression trees to predict normalized durations;
(7) training a first classifier to label accents and boundaries by:
(a) training an n-next-neighborhood classifier to recognize predicted accent and predicted boundary labels;
(b) training the refined classification and regression trees to output accent and boundary probabilities from linguistic features and relative syllable durations;
(c) relabeling the annotations;
(8) training the refined classification and regression trees to predict accents and boundaries from linguistic features only;
(9) relabeling the annotations; and
(10) returning to step (5) until prosodic labels stabilize.
2. The computing device of claim 1 , wherein the first module comprises classification and regression trees that generate initial accent and boundary labels by considering pauses and relative syllable durations.
3. The computing device of claim 2 , further comprising a module that predicts a plurality of fundamental frequency targets per syllable and that predicts a z-score for each phone, wherein the second module comprises classification and regression trees that predict three F0 targets per syllable.
4. The computing device of claim 2 , wherein the first module further makes initial accent labels applying a simple rule on text-derived features only.
5. The computing device of claim 1 , wherein pause durations and syllable durations, obtained from phonetic segmentation and normalization, are added to textual features in the annotated speech files.
6. The computing device of claim 1 , wherein the annotations in the annotated speech files relate to words, punctuation, pronunciation, word and syllable boundaries, lexical stress and parts of speech.
7. The computing device of claim 6 , wherein the computing device extracts F0 contours from the annotated speech files, interpolates for unvoiced regions, takes three samples per syllable, performs a cluster analysis, and adds quantized F0s to the annotations.
8. A method of generating a prosody model for generating synthetic speech from text-derived annotated speech files having annotations, the method comprising:
(1) generating initial accent and boundary labels by considering pauses and relative syllable durations in the annotated speech files based on binary decisions; and
(2) relabeling the annotations;
(3) returning to step (1) until prosodic labels stabilize;
(4) iteratively training classification and regression trees to predict durations and F0s from added predicted linguistic features and prosodic labels to yield refined classification and regressive trees;
(5) training the refined classification and regression trees to predict normalized durations;
(6) training a first classifier to label accents and boundaries by:
(a) training a classifier to recognize predicted accent and predicted boundary labels;
(b) training the refined classification and regression trees to output accent and boundary probabilities from linguistic features and relative syllable durations;
(c) relabeling the annotations;
(7) training the refined classification and regression trees to predict accents and boundaries from linguistic features only;
(8) relabeling the annotations; and
(9) returning to step (4) until prosodic labels stabilize.
9. The method of claim 8 , further comprising:
adding predicted linguistic features to text-derived annotations in the speech files;
adding normalized syllable durations to the annotations; and
adding a plurality of extracted acoustic features to the annotations.
10. The method of claim 9 , further comprising, to generate the plurality of extracted acoustic features:
extracting F0 contours from the annotated speech files;
interpolating in unvoiced regions;
taking three samples per syllable;
performing a cluster analysis; and
adding quantized F0s to the annotations.
11. The method of claim 10 , wherein the cluster analysis is performed to obtain a plurality of prototypes representing different shapes of the F0 contours.
12. The method of claim 10 , wherein the plurality of extracted features comprises eleven extracted features.
13. The method of claim 9 , wherein the added linguistic features relate to a yes-no question.
14. The method of claim 9 , wherein the annotations in the annotated speech files comprise words, punctuation, pronunciation, word and syllable boundaries, lexical stress and parts-of-speech.
15. A non-transitory computer readable medium storing instructions for controlling a computer device to perform a method of generating a prosody model from text-derived annotated speech files having annotations for use in prosody prediction, the method comprising:
(1) generating, via a processor, initial accent and boundary labels by considering pauses and relative syllable durations in the annotation speech files;
(2) iteratively training classification and regression trees to predict duration and F0s from added predicted linguistic features and prosodic labels until prosodic labels stabilize;
(3) training refined classification and regression trees to predict normalized durations;
(4) training label accents and boundaries by:
(a) training a classifier to recognize predicted accent and predicted boundary labels;
(b) training the refined classification and regression trees to output accent and boundary probabilities from linguistic features and relative syllable durations;
(c) relabeling the annotations;
(5) training the refined classification and regression trees to predict accents and boundaries from linguistic features only;
(6) relabeling the annotations; and
(7) returning to step (2) until prosodic labels stabilize.