Face pose estimation/three-dimensional face reconstruction method, apparatus, and electronic device
This application discloses methods, apparatus, and electronic devices for face pose estimation and three-dimensional face reconstruction. The face pose estimation method comprises: acquiring a two-dimensional face image for processing, constructing a three-dimensional face model corresponding to the two-dimensional face image, and determining a face pose of the two-dimensional face image based on face feature points of the three-dimensional face model and face feature points of the two-dimensional face image. With this approach, the face pose estimation is performed based on the three-dimensional face model corresponding to the two-dimensional face image, instead of only based on a three-dimensional average face model. As a result, a high accuracy pose estimation can be obtained even for a face with large-angle and exaggerated facial expressions. Thus, robustness of the pose estimation can be effectively improved.
1. A face pose estimation method, comprising:
acquiring a two-dimensional face image;
constructing a three-dimensional face model corresponding to the two-dimensional face image, wherein the constructing of the three-dimensional face model comprises:
obtaining a plurality of three-dimensional face model samples with a neutral facial expression;
applying a dimensionality reduction algorithm to the plurality of three-dimensional face model samples with the neutral facial expression to determine a three-dimensional average face model;
for each of the plurality of three-dimensional face model samples with the neutral facial expression, generating multiple face model samples corresponding to multiple non-neutral facial expressions, thereby obtaining a plurality of three-dimensional face model samples for each facial expression;
for each facial expression, determining a facial expression base by (1) obtaining an average facial expression model for the facial expression based on the plurality of three-dimensional face model samples for the facial expression; (2) subtracting the average facial expression model for the facial expression from each of the plurality of three-dimensional face model samples to obtain a difference data vector, (3) determining the difference data vector as the facial expression base for the facial expression;
determining a projection mapping matrix from the three-dimensional average face model to the two-dimensional face image based on internal face feature points of the two-dimensional face image and the three-dimensional average face model, wherein the internal face feature points include one or more of eyes, nose tip, mouth corner points, or eyebrows;
constructing a first three-dimensional face model corresponding to the two-dimensional face image based on the projection mapping matrix and feature vectors of a three-dimensional feature face space;
performing contour feature point fitting on the first three-dimensional face model based on face contour feature points of the two-dimensional face image;
after the contour feature point fitting, performing facial expression fitting on the first three-dimensional face model based on the internal face feature points of the two-dimensional face image and at least one of the facial expression bases;
determining an error between the two-dimensional face image and the three-dimensional face model; and
in response to the error being less than a preset error, adopting the first three-dimensional face model.
2. The method of claim 1 , wherein the constructing the three-dimensional face model corresponding to the two-dimensional face image comprises:
constructing the three-dimensional face model corresponding to the two-dimensional face image by using a face shape fitting algorithm.
3. The method of claim 1 , wherein the constructing the three-dimensional face model corresponding to the two-dimensional face image comprises:
constructing the three-dimensional face model corresponding to the two-dimensional face image by using a facial expression fitting algorithm.
4. The method of claim 1 , wherein the constructing the three-dimensional face model corresponding to the two-dimensional face image comprises:
constructing the three-dimensional face model corresponding to the two-dimensional face image by using a face shape and expression fitting algorithm.
5. The method of claim 1 , further comprising:
in response to the error being greater than or equal to the preset error, constructing the three-dimensional face model corresponding to the two-dimensional face image based on the three-dimensional face model fitted with the facial expression.
6. The method of claim 1 , wherein the performing the contour feature point fitting on the first three-dimensional face model based on the face contour feature points of the two-dimensional face image comprises:
selecting, from the first three-dimensional face model, three-dimensional points corresponding to the face contour feature points of the two-dimensional face image as initial three-dimensional contour feature points;
mapping the initial three-dimensional contour feature points to the two-dimensional face image by using the projection mapping matrix; and
selecting, by using a nearest neighbor matching algorithm, mapped three-dimensional points corresponding to two-dimensional contour feature points as face contour feature points of the first three-dimensional face model.
7. The method of claim 1 , further comprising:
determining at least one facial expression base corresponding to the three-dimensional average face model based on the plurality of three-dimensional face model samples.
8. A three-dimensional face reconstruction method, comprising:
acquiring a two-dimensional face image for processing;
applying a dimensionality reduction algorithm to a plurality of three-dimensional face model samples with a neutral facial expression to determine a three-dimensional average face model with the neutral facial expression;
for each of the plurality of three-dimensional face model samples with the neutral facial expression, generating multiple face model samples corresponding to multiple non-neutral facial expressions, thereby obtaining a plurality of three-dimensional face model samples for each facial expression;
for each facial expression, determining a facial expression base by (1) obtaining an average facial expression model for the facial expression based on the plurality of three-dimensional face model samples for the facial expression; (2) subtracting the average facial expression model for the facial expression from each of the plurality of three-dimensional face model samples to obtain a difference data vector, (3) determining the difference data vector as the facial expression base for the facial expression;
determining a projection mapping matrix from the three-dimensional average face model to the two-dimensional face image based on internal feature points of the two-dimensional face image and the three-dimensional average face model, wherein the internal face feature points include one or more of eyes, nose tip, mouth corner points, or eyebrows;
constructing a first three-dimensional face model corresponding to the two-dimensional face image based on the projection mapping matrix and feature vectors of a three-dimensional feature face space;
performing face contour feature points fitting on the three-dimensional average face model based on the two-dimensional face image;
after the contour feature point fitting, performing facial expression fitting on the first three-dimensional face model based on the internal face feature points of the two-dimensional face image and at least one of the facial expression bases;
determining an error between the two-dimensional face image and the fitted three-dimensional face model; and
in response to the error being less than a preset error, adopting the first three-dimensional face model.
9. The method of claim 8 ,
wherein after the constructing the first three-dimensional face model corresponding to the two-dimensional face image based on the projection mapping matrix and the feature vectors of the three-dimensional feature face space, the method further comprises:
performing expression fitting on the first three-dimensional face model based on at least one facial expression base corresponding to the three-dimensional average face model;
wherein the performing the contour feature point fitting on the first three-dimensional face model based on the face contour feature points of the two-dimensional face image comprises:
performing the contour feature point fitting on the first three-dimensional face model fitted with the facial expression based on the face contour feature points of the two-dimensional face image.
10. The method of claim 8 , wherein the performing the contour feature point fitting on the first three-dimensional face model based on the face contour feature points of the two-dimensional face image comprises:
selecting, from the first three-dimensional face model, three-dimensional points corresponding to the face contour feature points of the two-dimensional face image as initial three-dimensional contour feature points;
mapping the initial three-dimensional contour feature points to the two-dimensional face image by using the projection mapping matrix; and
selecting, by using a nearest neighbor matching algorithm, mapped three-dimensional points corresponding to two-dimensional contour feature points as face contour feature points of the first three-dimensional face model.
11. The method of claim 8 , further comprising:
determining at least one facial expression base corresponding to the three-dimensional average face model based on the plurality of three-dimensional face model samples.
12. An electronic device, comprising:
a processor; and
a memory, configured to store a program for implementing a face pose estimation method, wherein the device, after being powered on and running the program of the face pose estimation method through the processor, performs the following steps:
acquiring a two-dimensional face image;
constructing a three-dimensional face model corresponding to the two-dimensional face image, wherein the constructing of the three-dimensional face model comprises:
obtaining a plurality of three-dimensional face model samples with a neutral facial expression;
applying a dimensionality reduction algorithm to the plurality of three-dimensional face model samples with the neutral facial expression to determine a three-dimensional average face model;
for each of the plurality of three-dimensional face model samples with the neutral facial expression, generating multiple face model samples corresponding to multiple non-neutral facial expressions, thereby obtaining a plurality of three-dimensional face model samples for each facial expression;
for each facial expression, determining a facial expression base by (1) obtaining an average facial expression model for the facial expression based on the plurality of three-dimensional face model samples for the facial expression; (2) subtracting the average facial expression model for the facial expression from each of the plurality of three-dimensional face model samples to obtain a difference data vector, (3) determining the difference data vector as the facial expression base for the facial expression;
determining a projection mapping matrix from the three-dimensional average face model to the two-dimensional face image based on internal face feature points of the two-dimensional face image and the three-dimensional average face model, wherein the internal face feature points include one or more of eyes, nose tip, mouth corner points, or eyebrows;
constructing a first three-dimensional face model corresponding to the two-dimensional face image based on the projection mapping matrix and feature vectors of a three-dimensional feature face space;
performing contour feature point fitting on the first three-dimensional face model based on face contour feature points of the two-dimensional face image;
after the contour feature point fitting, performing facial expression fitting on the first three-dimensional face model based on the internal face feature points of the two-dimensional face image and at least one of the facial expression bases;
determining an error between the two-dimensional face image and the three-dimensional face model; and
in response to the error being less than a preset error, adopting the first three-dimensional face model.
13. The electronic device of claim 12 , wherein the constructing the three-dimensional face model corresponding to the two-dimensional face image comprises:
constructing the three-dimensional face model corresponding to the two-dimensional face image by using a face shape fitting algorithm.