Device and method for generating speech animation

1. A method for generating speech animation from an audio signal, comprising:

receiving the audio signal;

transforming the received audio signal into frequency-domain audio features including:

calculating mel-frequency cepstral coefficients (MFCCs) of the audio signal;

calculating a first and a second derivative components of the MFCCs and aggregating the first and second derivative components into a single audio feature frame; and

generating the audio features by aggregating the single audio feature frame with forward and backward contextual vectors of the single audio feature frame;

performing neural-network processing on the frequency-domain audio features to recognize phonemes, wherein the neural-network processing is performed using a neural network trained with a phoneme dataset comprising audio signals with corresponding ground-truth phoneme labels, including:

applying convolutional neural network (CNN) filters on the audio features to generate a plurality of sets of intermediate features, each set of intermediate features corresponding to one of a plurality of CNN output channels;

applying parallel temporal modeling on the intermediate features using a plurality of long short-term memory (LSTM) networks to generate LSTM stack block (LSB) output features, wherein each LSTM network is applied to one set of intermediate features to generate a set of LSB output features;

applying unified temporal modeling using a unified LSTM network to the LSB output features to generate unified LSTM output features;

applying a fully connected layer (FCL) to the unified LSTM output features to generate hidden Markov models (HMM) tied states; and

applying an HMM tri-phone decoder to the HMM tied-states to generate recognized phoneme labels; and

generating the speech animation from the recognized phonemes.

2. The method according to claim 1 , further comprising:

storing the audio features in a first storage buffer;

storing the HMM tied states in a second storage buffer; and

storing the recognized phonemes in a third storage buffer;

wherein the first storage buffer, the second storage buffer, and the third storage buffer are fixed size first in first out (FIFO) buffers.

3. The method according to claim 1 , further comprising:

generating a first combined feature by combining the single audio feature frame with the LSB output features; and

using the first combined feature as input to the unified LSTM network to generate the unified LSTM output features.

4. The method according to claim 1 , further comprising:

generating a second combined feature by combining the LSB output features with the unified LSTM output features; and

using the second combined feature as input to the FCL to generate the HMM tied states.

5. The method according to claim 1 , wherein generating the speech animation from the recognized phonemes includes:

generating a 2-dimensional (2D) viseme field and dividing the 2D viseme field into a plurality of blocks, wherein each recognized phoneme corresponds to a block of the 2D viseme field.

6. A device for generating speech animation from an audio signal, comprising:

a memory, storing computer-executable instructions; and

a processor, coupled with the memory and, when the computer-executable instructions being executed, configured to:

receive the audio signal;

transform the received audio signal into frequency-domain audio features;

calculate mel-frequency cepstral coefficients (MFCCs) of the audio signal;

calculate a first and a second derivative components of the MFCCs and aggregating the first and second derivative components into a single audio feature frame;

generate the audio features by aggregating the single audio feature frame with forward and backward contextual vectors of the single audio feature frame;

perform neural-network processing on the frequency-domain audio features to recognize phonemes, wherein the neural-network processing is performed using a neural network trained with a phoneme dataset comprising audio signals with corresponding ground-truth phoneme labels;

apply convolutional neural network (CNN) filters on the audio features to generate a plurality of sets of intermediate features, each set of intermediate features corresponding to one of a plurality of CNN output channels;

apply parallel temporal modeling on the intermediate features using a plurality of long short-term memory (LSTM) networks to generate LSTM stack block (LSB) output features, wherein each LSTM network is applied to one set of intermediate features to generate a set of LSB output features;

apply unified temporal modeling using a unified LSTM network to the LSB output features to generate unified LSTM output features;

apply a fully connected layer (FCL) to the unified LSTM output features to generate hidden Markov models (HMM) tied states;

apply an HMM tri-phone decoder to the HMM tied-states to generate recognized phoneme labels; and

generate the speech animation from the recognized phonemes.

7. The device according to claim 6 , wherein the processor is further configured to:

store the audio features in a first storage buffer;

store the HMM tied states in a second storage buffer; and

store the recognized phonemes in a third storage buffer;

wherein the first storage buffer, the second storage buffer, and the third storage buffer are fixed size first in first out (FIFO) buffers.

8. The device according to claim 6 , wherein the processor is further configured to:

generate a first combined feature by combining the single audio feature frame with the LSB output features; and

use the first combined feature as input to the unified LSTM network to generate the unified LSTM output features.

9. The device according to claim 6 , wherein the processor is further configured to:

generate a second combined feature by combining the LSB output features with the unified LSTM output features; and

use the second combined feature as input to the FCL to generate the HMM tied states.

10. The device according to claim 6 , wherein the processor is further configured to:

generate a 2-dimensional (2D) viseme field and dividing the 2D viseme field into a plurality of blocks, wherein each recognized phoneme corresponds to a block of the 2D viseme field.

11. A non-transitory computer-readable storage medium storing a plurality of instructions, wherein when the plurality of instructions are executed by a processor, cause the processor to:

receive the audio signal;

transform the received audio signal into frequency-domain audio features;

calculate mel-frequency cepstral coefficients (MFCCs) of the audio signal;

calculate a first and a second derivative components of the MFCCs and aggregating the first and second derivative components into a single audio feature frame; and

generate the audio features by aggregating the single audio feature frame with forward and backward contextual vectors of the single audio feature frame;

perform neural-network processing on the frequency-domain audio features to recognize phonemes, wherein the neural-network processing is performed using a neural network trained with a phoneme dataset comprising audio signals with corresponding ground-truth phoneme labels;

apply convolutional neural network (CNN) filters on the audio features to generate a plurality of sets of intermediate features, each set of intermediate features corresponding to one of a plurality of CNN output channels;

apply parallel temporal modeling on the intermediate features using a plurality of long short-term memory (LSTM) networks to generate LSTM stack block (LSB) output features, wherein each LSTM network is applied to one set of intermediate features to generate a set of LSB output features;

apply unified temporal modeling using a unified LSTM network to the LSB output features to generate unified LSTM output features;

apply a fully connected layer (FCL) to the unified LSTM output features to generate hidden Markov models (HMM) tied states;

apply an HMM tri-phone decoder to the HMM tied-states to generate recognized phoneme labels; and

generate the speech animation from the recognized phonemes.

12. The non-transitory computer-readable storage medium according to claim 11 , wherein the plurality of instructions further cause the processor to:

store the audio features in a first storage buffer;

store the HMM tied states in a second storage buffer; and

store the recognized phonemes in a third storage buffer;

wherein the first storage buffer, the second storage buffer, and the third storage buffer are fixed size first in first out (FIFO) buffers.

13. The non-transitory computer-readable storage medium according to claim 11 , wherein the plurality of instructions further cause the processor to:

generate a first combined feature by combining the single audio feature frame with the LSB output features; and

use the first combined feature as input to the unified LSTM network to generate the unified LSTM output features.

14. The non-transitory computer-readable storage medium according to claim 11 , wherein the plurality of instructions further cause the processor to:

generate a second combined feature by combining the LSB output features with the unified LSTM output features; and

use the second combined feature as input to the FCL to generate the HMM tied states.