Method and device for three-dimensional modeling
The present invention provides a method and a device for three-dimensional modeling, used in the field of image processing. The method includes: receiving a first two-dimensional image and a depth map corresponding to the first two-dimensional image; the first two-dimensional image and the depth map respectively include a face; fitting three-dimensional face model by a first three-dimensional face database according to a position of a face feature point in the first two-dimensional image and the depth map corresponding to the first two-dimensional image. The method and the device for three-dimensional modeling provided by the present invention can improve the efficiency of the facial three-dimensional modeling and the precision of modeling.
1. A method for three-dimensional modeling, comprising:
receiving a first two-dimensional image and a depth map corresponding to the first two-dimensional image, wherein the first two-dimensional image and the depth map respectively comprises a face; and
fitting a three-dimensional face model by a first three-dimensional face database according to a position of a face feature point in the first two-dimensional image and the depth map corresponding to the first two-dimensional image.
2. The method according to claim 1 , wherein
the first three-dimensional face database comprises an average face model and at least one of the following: a shape feature vector and an expression feature vector.
3. The method according to claim 2 , wherein
the step of fitting the three-dimensional face model by the first three-dimensional face database according to the position of the face feature point in the first two-dimensional image and the depth map corresponding to the first two-dimensional image comprises:
estimating an initial transformation matrix of a three-dimensional point cloud model corresponding to the depth map from the average face model, according to the initial transformation matrix and at least one of the following: the shape feature vector and the expression feature vector, and aiming at a first condition, fitting the three-dimensional face model from the average face model.
4. The method according to claim 3 , wherein the first condition comprises at least one of the following:
a distance between a projection position of a feature point of the three-dimensional face model in an image coordinate system and the position of the feature point of the first two-dimensional image corresponding to the feature point of the three-dimensional face model is smallest; and
a distance between the three-dimensional face model and a point pair corresponding to the three-dimensional point cloud mapped by the depth map is smallest.
5. The method according to claim 3 , wherein
the step of estimating the initial transformation matrix of the three-dimensional point cloud model corresponding to the depth map by the average face model comprises:
calculating a three-dimensional position of the face feature point according to the position of the face feature point in the first two-dimensional image and the depth map corresponding to the first two-dimensional image; and
estimating the initial transformation matrix of the three-dimensional point cloud model corresponding to the depth map from the average face model according to the three-dimensional position of the face feature point and the three-dimensional position of the feature point of the average face model.
6. A device for three-dimensional modeling, comprising:
a receiver, configured to receive a first two-dimensional image and a depth map corresponding to the first two-dimensional image, wherein the first two-dimensional image and the depth map respectively comprise a face; and
a processor, configured to fit a three-dimensional face model by a first three-dimensional face database according to a position of a face feature point in the first two-dimensional image and the depth map corresponding to the first two-dimensional image.
7. The device according to claim 6 , wherein
the first three-dimensional face database comprises an average face model and at least one of the following: a shape feature vector and an expression feature vector.
8. The device according to claim 7 , wherein
the processor is specifically configured to:
estimate an initial transformation matrix of a three-dimensional point cloud model corresponding to the depth map from the average face model, according to the initial transformation matrix and at least one of the following: the shape feature vector and the expression feature vector, and aiming at a first condition, fitting the three-dimensional face model from the average face model.
9. The device according to claim 8 , wherein the first condition comprises at least one of the following:
a distance between a projection position of a feature point of the three-dimensional face model in the image coordinate system and the position of the feature point of the first two-dimensional image corresponding to the feature point of the three-dimensional face model is smallest; and
a distance between the three-dimensional face model and a point pair corresponding to the three-dimensional point cloud mapped by the depth map is smallest.
10. The device according to claim 8 , wherein the processor is specifically configured to:
calculate a three-dimensional position of the face feature point according to the position of the face feature point in the first two-dimensional image and the depth map corresponding to the first two-dimensional image; and
estimate the initial transformation matrix of the three-dimensional point cloud model corresponding to the depth map from the average face model according to the three-dimensional position of the face feature point and the three-dimensional position of the feature point of the average face model.
11. A computer readable storage medium, storing a computer program, wherein when the computer program is executed by a first processor, the following steps are performed:
receiving a first two-dimensional image and a depth map corresponding to the first two-dimensional image, wherein the first two-dimensional image and the depth map respectively comprises a face; and
fitting a three-dimensional face model by a first three-dimensional face database according to a position of a face feature point in the first two-dimensional image and the depth map corresponding to the first two-dimensional image.
12. A device for three-dimensional modeling, comprising a memory, a second processor, and a computer program stored in the memory and operated by the second processor, wherein when the computer program is executed by the second processor, the following steps are performed:
receiving a first two-dimensional image and a depth map corresponding to the first two-dimensional image, wherein the first two-dimensional image and the depth map respectively comprises a face; and
fitting a three-dimensional face model by a first three-dimensional face database according to a position of a face feature point in the first two-dimensional image and the depth map corresponding to the first two-dimensional image.
13. The method according to claim 4 , wherein
the step of estimating the initial transformation matrix of the three-dimensional point cloud model corresponding to the depth map by the average face model comprises:
calculating a three-dimensional position of the face feature point according to the position of the face feature point in the first two-dimensional image and the depth map corresponding to the first two-dimensional image; and
estimating the initial transformation matrix of the three-dimensional point cloud model corresponding to the depth map from the average face model according to the three-dimensional position of the face feature point and the three-dimensional position of the feature point of the average face model.
14. The device according to claim 9 , wherein the processor is specifically configured to:
calculate a three-dimensional position of the face feature point according to the position of the face feature point in the first two-dimensional image and the depth map corresponding to the first two-dimensional image; and
estimate the initial transformation matrix of the three-dimensional point cloud model corresponding to the depth map from the average face model according to the three-dimensional position of the face feature point and the three-dimensional position of the feature point of the average face model.
15. The computer readable storage medium according to claim 11 , wherein
the first three-dimensional face database comprises an average face model and at least one of the following: a shape feature vector and an expression feature vector.
16. The computer readable storage medium according to claim 11 , wherein
the step of fitting the three-dimensional face model by the first three-dimensional face database according to the position of the face feature point in the first two-dimensional image and the depth map corresponding to the first two-dimensional image comprises:
estimating an initial transformation matrix of a three-dimensional point cloud model corresponding to the depth map from the average face model; according to the initial transformation matrix and at least one of the following: the shape feature vector and the expression feature vector, and aiming at a first condition, fitting the three-dimensional face model from the average face model.
17. The computer readable storage medium according to claim 16 , wherein
the first condition comprises at least one of the following:
a distance between a projection position of a feature point of the three-dimensional face model in an image coordinate system and the position of the feature point of the first two-dimensional image corresponding to the feature point of the three-dimensional face model is smallest; and
a distance between the three-dimensional face model and a point pair corresponding to the three-dimensional point cloud mapped by the depth map is smallest.
18. The computer readable storage medium according to claim 16 , wherein
the step of estimating the initial transformation matrix of the three-dimensional point cloud model corresponding to the depth map by the average face model comprises:
calculating a three-dimensional position of the face feature point according to the position of the face feature point in the first two-dimensional image and the depth map corresponding to the first two-dimensional image; and
estimating the initial transformation matrix of the three-dimensional point cloud model corresponding to the depth map from the average face model according to the three-dimensional position of the face feature point and the three-dimensional position of the feature point of the average face model.
19. The computer readable storage medium according to claim 17 , wherein
the step of estimating the initial transformation matrix of the three-dimensional point cloud model corresponding to the depth map by the average face model comprises:
calculating a three-dimensional position of the face feature point according to the position of the face feature point in the first two-dimensional image and the depth map corresponding to the first two-dimensional image; and
estimating the initial transformation matrix of the three-dimensional point cloud model corresponding to the depth map from the average face model according to the three-dimensional position of the face feature point and the three-dimensional position of the feature point of the average face model.
20. The device for three-dimensional modeling according to claim 12 , wherein
the first three-dimensional face database comprises an average face model and at least one of the following: a shape feature vector and an expression feature vector.
21. The device for three-dimensional modeling according to claim 20 , wherein
the step of fitting the three-dimensional face model by the first three-dimensional face database according to the position of the face feature point in the first two-dimensional image and the depth map corresponding to the first two-dimensional image comprises:
estimating an initial transformation matrix of a three-dimensional point cloud model corresponding to the depth map from the average face model; according to the initial transformation matrix and at least one of the following: the shape feature vector and the expression feature vector, and aiming at a first condition, fitting the three-dimensional face model from the average face model.
22. The device for three-dimensional modeling according to claim 21 , wherein
the first condition comprises at least one of the following:
a distance between a projection position of a feature point of the three-dimensional face model in an image coordinate system and the position of the feature point of the first two-dimensional image corresponding to the feature point of the three-dimensional face model is smallest; and
a distance between the three-dimensional face model and a point pair corresponding to the three-dimensional point cloud mapped by the depth map is smallest. 18 . The computer readable storage medium according to claim 11 , storing a computer program, wherein the steps of the claim 5 is performed when the computer program is executed by a first processor.
23. The device for three-dimensional modeling according to claim 21 , wherein
the step of estimating the initial transformation matrix of the three-dimensional point cloud model corresponding to the depth map by the average face model comprises:
calculating a three-dimensional position of the face feature point according to the position of the face feature point in the first two-dimensional image and the depth map corresponding to the first two-dimensional image; and
estimating the initial transformation matrix of the three-dimensional point cloud model corresponding to the depth map from the average face model according to the three-dimensional position of the face feature point and the three-dimensional position of the feature point of the average face model.
24. The device for three-dimensional modeling according to claim 22 , wherein
the step of estimating the initial transformation matrix of the three-dimensional point cloud model corresponding to the depth map by the average face model comprises:
calculating a three-dimensional position of the face feature point according to the position of the face feature point in the first two-dimensional image and the depth map corresponding to the first two-dimensional image; and
estimating the initial transformation matrix of the three-dimensional point cloud model corresponding to the depth map from the average face model according to the three-dimensional position of the face feature point and the three-dimensional position of the feature point of the average face model.