Hybrid predictive model for enhancing prosodic expressiveness
View Patent ↗Systems and methods for prosody prediction include extracting features from runtime data using a parametric model. The features from runtime data are compared with features from training data using an exemplar-based model to predict prosody of the runtime data. The features from the training data are paired with exemplars from the training data and stored on a computer readable storage medium.
1. A method for prosody prediction, comprising:
extracting features from runtime data using a parametric model;
comparing the features from runtime data with features from training data using an exemplar-based model to predict prosody of the runtime data, the features from the training data being paired with exemplars from the training data and stored on a computer readable storage medium; and
synthesizing speech, by a speech synthesizer, using the predicted prosody,
wherein the parametric model includes a plurality of layers, and the features include deep layer features that, in turn, include features after data has been transformed up to a layer of the parametric model before an output layer, and
wherein a hybrid model including the parametric model and the exemplar-based model is built for each genre of interest.
2. The method as recited in claim 1 , wherein the parametric model includes a neural network model and the exemplar-based model includes a Gaussian Process model.
3. The method as recited in claim 1 , wherein the features include deep layer features.
4. The method as recited in claim 1 , further comprising training the parametric model to transform the training data to reproduce training targets.
5. The method as recited in claim 1 , further comprising training the exemplar-based model to determine exemplars from the training data.
6. The method as recited in claim 1 , further comprising constraining a number of nodes in a deepest layer of the parametric model before an output layer to reduce dimensionality.
7. The method as recited in claim 1 , wherein a hybrid model including the parametric model and the exemplar-based model is built for all genres of interest by pooling data from the genres of interest.
8. The method as recited in claim 7 , wherein the hybrid model includes a genre-indictor feature to trigger appropriate deep layer features.
9. The method as recited in claim 1 , wherein the parametric model is trained to apply transformations to training data to reproduce a training target, and the features are extracted one layer prior to the training target.
10. The method as recited in claim 1 , wherein the features extracted one layer prior to the training target are the features with training data that are compared to the features from runtime data to predict the prosody of the runtime data.
11. The method as recited in claim 1 , wherein exemplars of the exemplar-based model are determined from the transformed training data and the training target, and the exemplars are paired with the extracted features for comparison to the features from the runtime data.
12. The method of claim 1 , wherein the plurality of layers include n transforming layers, wherein each of the n transforming layers computes a weighted sum of outputs of a preceding layer, followed by a non-linearity.
13. The method of claim 1 , wherein the plurality of layers include n transforming layers, and the deep layer features, in turn, include the features after the data has been transformed up to an n−1 transforming layer of the parametric model.
14. The method of claim 13 , wherein raw numerical inputs are passed through the first n−1 transforming layers of the parametric model.
15. The method of claim 1 , wherein the deep layer features comprise prosody prediction features.
16. The method of claim 15 , wherein the prosody prediction features comprise phonetic identity and syllable count.
17. A method for prosody prediction, comprising:
extracting features from runtime data using a parametric model;
comparing the features from runtime data with features from training data using an exemplar-based model to predict prosody of the runtime data, the features from the training data being paired with exemplars from the training data and stored on a computer readable storage medium; and
synthesizing speech, by a speech synthesizer, using the predicted prosody,
wherein the parametric model includes a plurality of layers, and the features include deep layer features that, in turn, include features after data has been transformed up to a layer of the parametric model before an output layer, and
wherein a hybrid model including the parametric model and the exemplar-based model is built for all genres of interest by pooling data from hybrid models built for each of two or more genres of interest.