Method and apparatus for predicting mouth-shape feature, and electronic device
A method and apparatus for predicting a mouth-shape feature, and an electronic device are provided. A specific implementation of the method comprises: recognizing a phonetic posterior gram (PPG) of a phonetic feature; and performing a prediction on the PPG by using a neural network model, to predict a mouth-shape feature of the phonetic feature, the neural network model being obtained by training with training samples and an input thereof including a PPG and an output thereof including a mouth-shape feature, and the training samples including a PPG training sample and a mouth-shape feature training sample.
1. A method for predicting a mouth-shape feature, comprising:
recognizing a phonetic posterior gram (PPG) of a phonetic feature of a speech; and
performing a prediction on the PPG by using a neural network model, to predict a mouth-shape feature of the phonetic feature, the neural network model being obtained by training with training samples and an input thereof including a PPG and an output thereof including a mouth-shape feature, and the training samples including a PPG training sample and a mouth-shape feature training sample,
wherein the PPG training sample comprises:
PPGs of target phonetic features, a target phonetic feature being a phonetic feature having complete semantics which is obtained by slicing according to the semantics of the speech; and
the mouth-shape feature training sample comprises:
mouth-shape features corresponding to the PPGs of the target phonetic features,
wherein the neural network model is a multi-branch neural network model, and the mouth-shape feature of the phonetic feature includes at least two of:
a regression mouth-shape point, a mouth-shape thumbnail, a blend shape coefficient, or three-dimensional morphable models (3DMM) expression coefficient,
wherein the multi-branch neural network model has a plurality of branch networks, and each branch network predicts a mouth-shape feature of the regression mouth-shape point, the mouth-shape thumbnail, the blend shape coefficient, and the 3DMM expression coefficient.
2. The method according to claim 1 , wherein a frequency of a target phonetic feature matches a frequency of a mouth-shape feature corresponding to a PPG of the target phonetic feature.
3. The method according to claim 1 , wherein the multi-branch neural network model is a recurrent neural network (RNN) model having an autoregressive mechanism, and a process of training the RNN model includes:
performing the training by using a mouth-shape feature training sample of a frame preceding a current frame as an input, by using a PPG training sample of the current frame as a condition constraint, and a mouth-shape feature training sample of the current frame as a target.
4. The method according to claim 1 , further comprising:
performing predictions on PPGs of pieces of pieces of real speech data using the neural network model, to obtain mouth-shape features of the pieces of real speech data; and
constructing a mouth-shape feature index library based on the mouth-shape features of the pieces of real speech data, the mouth-shape feature index library being used for synthesizing a mouth shape of a virtual image.
5. An electronic device, comprising:
at least one processor; and
a memory, communicated with the at least one processor,
wherein the memory stores an instruction executable by the at least one processor, and the instruction, when executed by the at least one processor, causes the at least one processor to perform operations, the operations comprise:
recognizing a phonetic posterior gram (PPG) of a phonetic feature of a speech; and
performing a prediction on the PPG by using a neural network model, to predict a mouth-shape feature of the phonetic feature, the neural network model being obtained by training with training samples and an input thereof including a PPG and an output thereof including a mouth-shape feature, and the training samples including a PPG training sample and a mouth-shape feature training sample,
wherein the PPG training sample comprises:
PPGs of target phonetic features, a target phonetic feature being a phonetic feature having complete semantics which is obtained by slicing according to the semantics of the speech; and
the mouth-shape feature training sample comprises:
mouth-shape features corresponding to the PPGs of the target phonetic features,
wherein the neural network model is a multi-branch neural network model, and the mouth-shape feature of the phonetic feature includes at least two of:
a regression mouth-shape point, a mouth-shape thumbnail, a blend shape coefficient, or three-dimensional morphable models (3DMM) expression coefficient,
wherein the multi-branch neural network model has a plurality of branch networks, and each branch network predicts a mouth-shape feature of the regression mouth-shape point, the mouth-shape thumbnail, the blend shape coefficient, and the 3DMM expression coefficient.
6. The electronic device according to claim 5 , wherein a frequency of a target phonetic feature matches a frequency of a mouth-shape feature corresponding to a PPG of the target phonetic feature.
7. The electronic device according to claim 5 , wherein the multi-branch neural network model is a recurrent neural network (RNN) model having an autoregressive mechanism, and a process of training the RNN model includes:
performing the training by using a mouth-shape feature training sample of a frame preceding a current frame as an input, by using a PPG training sample of the current frame as a condition constraint, and a mouth-shape feature training sample of the current frame as a target.
8. The electronic device according to claim 5 , wherein the operations further comprise:
performing predictions on PPGs of pieces of real speech data using the neural network model, to obtain mouth-shape features of the pieces of real speech data; and
constructing a mouth-shape feature index library based on the mouth-shape features of the pieces of real speech data, the mouth-shape feature index library being used for synthesizing a mouth shape of a virtual image.
9. A non-transitory computer readable storage medium, storing a computer instruction, wherein the computer instruction, when executed by a processor, causes the processor to perform operations, the operations comprise:
recognizing a phonetic posterior gram (PPG) of a phonetic feature of a speech; and
performing a prediction on the PPG by using a neural network model, to predict a mouth-shape feature of the phonetic feature, the neural network model being obtained by training with training samples and an input thereof including a PPG and an output thereof including a mouth-shape feature, and the training samples including a PPG training sample and a mouth-shape feature training sample,
wherein the PPG training sample comprises:
PPGs of target phonetic features, a target phonetic feature being a phonetic feature having complete semantics which is obtained by slicing according to the semantics of the speech; and
the mouth-shape feature training sample comprises:
mouth-shape features corresponding to the PPGs of the target phonetic features,
wherein the neural network model is a multi-branch neural network model, and the mouth-shape feature of the phonetic feature includes at least two of:
a regression mouth-shape point, a mouth-shape thumbnail, a blend shape coefficient, or three-dimensional morphable models (3DMM) expression coefficient,
wherein the multi-branch neural network model has a plurality of branch networks, and each branch network predicts a mouth-shape feature of the regression mouth-shape point, the mouth-shape thumbnail, the blend shape coefficient, and the 3DMM expression coefficient.
10. The medium according to claim 9 , wherein a frequency of a target phonetic feature matches a frequency of a mouth-shape feature corresponding to a PPG of the target phonetic feature.
11. The medium according to claim 9 , wherein the multi-branch neural network model is a recurrent neural network (RNN) model having an autoregressive mechanism, and a process of training the RNN model includes:
performing the training by using a mouth-shape feature training sample of a frame preceding a current frame as an input, by using a PPG training sample of the current frame as a condition constraint, and a mouth-shape feature training sample of the current frame as a target.
12. The medium according to claim 9 , wherein the operations further comprise:
performing predictions on PPGs of pieces of pieces of real speech data using the neural network model, to obtain mouth-shape features of the pieces of real speech data; and
constructing a mouth-shape feature index library based on the mouth-shape features of the pieces of real speech data, the mouth-shape feature index library being used for synthesizing a mouth shape of a virtual image.