Real-time generation of speech animation
To realistically animate a String (such as a sentence) a hierarchical search algorithm is provided to search for stored examples (Animation Snippets) of sub-strings of the String, in decreasing order of sub-string length, and concatenate retrieved sub-strings to complete the String of speech animation. In one embodiment, real-time generation of speech animation uses model visemes to predict the animation sequences at onsets of visemes and a look-up table based (data-driven) algorithm to predict the dynamics at transitions of visemes. Specifically posed Model Visemes may be blended with speech animation generated using another method at corresponding time points in the animation when the visemes are to be expressed. An Output Weighting Function is used to map Speech input and Expression input into Muscle-Based Descriptor weightings.
1. A method for animating a communicative utterance comprising:
receiving a String to be animated, the String comprising a plurality of communicative utterance atoms;
receiving a plurality of Collections, each Collection including a plurality of Items comprising unique atom strings, each Collection storing Items of different lengths, wherein at least one of the Items is based on an occurrence of acoustic silence before or after a particular communicative utterance atom, and each Item including at least one Animation Snippet of the Item;
hierarchically searching the Collections for Items matching substrings of the String, wherein the hierarchical searching favours longer Items;
retrieving Animation Snippets for matched Items to cover the communicative utterance atoms; and
combining the retrieved Animation Snippets to animate the String.
2. The method of claim 1 wherein the communicative utterance is speech; and
wherein the at least one of the Items is pre-defined as the occurrence of acoustic silence before or after the particular communicative utterance atom.
3. The method of claim 1 wherein at least one Item includes a plurality of Animation Snippets, and an Animation Snippet is retrieved based on its duration.
4. The method of claim 1 wherein at least one Item includes a plurality of Animation Snippets, and an Animation Snippet is retrieved based on corresponding speech features.
5. The method of claim 1 wherein the Animation Snippets are associated with sound corresponding to the animation.
6. The method of claim 5 further comprising:
compressing and/or stretching the Animation Snippets to match the sound corresponding to the animation.
7. The method of claim 1 wherein one or more of the Items are included in a first Collection of Items from a plurality of different Collections of Items;
wherein the first Collection of Items comprises a Sentence Boundary Collection that includes a first type of dipho string and a second type of dipho string;
wherein the first type of dipho string represents an occurrence of acoustic silence pre-defined as being paired with a subsequent respective syllable type; and
wherein the second type of dipho string represents an occurrence of acoustic silence pre-defined as being paired with a preceding respective syllable type.
8. The method of claim 1 wherein the Animation Snippets store Muscle-Based Descriptor weights.
9. A method for animating a phoneme in context comprising:
receiving a Model Viseme;
receiving an Animation Snippet corresponding to a time series of animation weights of the phoneme being pronounced in context, the Animation Snippet of the phoneme based at least on a string that includes an occurrence of acoustic silence pre-defined as being before or after at least a particular portion of the phoneme; and
blending between the animation weights of the Model Viseme and animation weights of the Animation Snippet to animate the phoneme in context, wherein a degree of blending of the Model Viseme over time is modelled by a function, with a peak of the function being at or about a peak of the phoneme being pronounced.
10. The method of claim 9 wherein the Model Viseme is a Lip-Readable Viseme depicting the phoneme in a lip-readable manner.
11. The method of claim 9 wherein the Model Viseme is represented as an animation sequence.
12. A method for expressive speech animation including:
receiving a Priority Weighting for one or more animation inputs, wherein an Output Weighting Function is a function of the Priority Weighting, the one more animation inputs based on concatenated Animation Snippets, wherein a first Animation Snippet represents a phoneme based at least on a string that includes an occurrence of acoustic silence pre-defined as being before or after at least a particular portion of the phoneme;
receiving a first animation input associated with muscle-based descriptor information and a second animation input associated with muscle-based descriptor information;
defining at least one Muscle-Based Descriptor Class Weighting for each muscle-based descriptor;
using the first animation input and the second animation input as arguments in the Output Weighting Function configured to map the animation inputs to muscle-based descriptor weightings for animating the expressive speech animation,
wherein the Output Weighting Function is a function of the Muscle-Based Descriptor Class Weightings,
wherein the Output Weighting Function is configured to reconcile muscle-based descriptor information from the first and second animation inputs; and
animating using the mapped muscle-based descriptor weightings.
13. The method of claim 12 wherein the first animation input is for animating Speech.
14. The method of claim 12 wherein the second animation input is for animating Expression.