Methods and software for generating a derived 3D object model from a single 2D image
Methods of reconstructing a 3D object model from portions of an object in a single 2D image using an existing 3D object model that is generic to a class to which the object in the 2D image belongs. In some embodiments, the methods determine a camera center from a computed estimated camera projection matrix. Landmark points on the object in the 2D image are back-projected to the existing 3D object model to warp the existing 3D object model according to the landmark points in the 2D image to create a derived 3D object model of the 2D object in the 2D image. Optional additional processing steps can include regularization, texturing, and infill of missing regions of the derived 3D object model. An example implementation of the disclosed methods is 3D craniofacial reconstruction from a single 2D image containing at least a portion of a face.
1. A method of generating a derived 3D object model of an object from an input 2D image containing at least a portion of the object, the method comprising:
receiving the input image and storing the input image in computer memory;
receiving a plurality of 2D landmark points on the object within the input 2D image;
executing an estimation algorithm to compute an estimated camera projection matrix for the input 2D image, wherein the estimation algorithm uses an existing 3D object model and is configured to minimize an average reprojection error;
generating a set of derived 3D landmark points by:
executing a camera center algorithm that computes a camera center from the estimated camera projection matrix; and
executing a 3D landmark algorithm that computes, for each of the plurality of 2D landmark points, a corresponding derived 3D landmark point;
warping the existing 3D object model using at least the derived 3D landmark points to generate the derived 3D object model; and
storing the derived 3D object model in the computer memory.
2. The method according to claim 1 , wherein the 3D landmark algorithm computes each of the derived 3D landmark points by finding a 3D point along a ray extending from the camera center through a corresponding one of the plurality of 2D landmark points that minimizes error between the 3D point and a corresponding 3D landmark point on the existing 3D object model.
3. The method according to claim 1 , wherein:
said receiving a plurality of 2D landmark points includes receiving at least six 2D facial landmark points, wherein at least four 2D facial landmark points of the at least six 2D facial landmark points are non-colinear; and
said warping the existing 3D object model includes warping the existing 3D object model using a thin plate spline warping function to move points on the existing 3D object model other than the 3D landmark points corresponding to the derived 3D landmark points.
4. The method according to claim 1 , further comprising regularizing the derived 3D object model by executing a regularization algorithm configured to perform data completion technique.
5. The method according to claim 1 , wherein the existing 3D object model comprises a generic model of the object.
6. The method according to claim 1 , wherein the object comprises a human head, the portion of the object comprises a human face, the derived 3D object model comprises a 3D craniofacial model, the plurality of 2D landmarks comprises a plurality of 2D facial landmarks, and the existing 3D object model comprises an existing 3D craniofacial model.
7. The method according to claim 6 , wherein the 3D landmark algorithm computes each of the derived 3D landmark points by finding a 3D point along a ray extending from the camera center through a corresponding one of the plurality of 2D facial landmark points that minimizes error between the 3D point and a corresponding 3D landmark point on the existing 3D craniofacial model.
8. The method according to claim 6 , wherein:
said receiving a plurality of 2D landmark points includes receiving at least six 2D facial landmark points, wherein at least four 2D facial landmark points of the at least six 2D facial landmark points are non-colinear; and
said warping the existing 3D craniofacial model includes warping the existing 3D craniofacial model using a thin plate spline warping function to move points on the existing 3D craniofacial model other than the 3D landmark points corresponding to the derived 3D landmark points.
9. The method according to claim 6 , further comprising regularizing the derived 3D craniofacial model by executing a regularization algorithm configured to perform data completion technique.
10. The method according to claim 1 , further comprising executing a camera projection matrix algorithm that minimizes an average reprojection error.
11. A machine-readable storage medium containing computer-executable instructions that, when executed on a computing system, performs a method of generating a derived 3D object model of an object from an input 2D image containing at least a portion of the object, the method comprising:
receiving the input image and storing the input image in computer memory;
receiving a plurality of 2D landmark points on the object within the input 2D image;
executing an estimation algorithm to compute an estimated camera projection matrix for the input 2D image, wherein the estimation algorithm uses an existing 3D object model and is configured to minimize an average reprojection error;
generating a set of derived 3D landmark points by:
executing a camera center algorithm that computes a camera center from the estimated camera projection matrix; and
executing a 3D landmark algorithm that computes, for each of the plurality of 2D landmark points, a corresponding derived 3D landmark point;
warping the existing 3D object model using at least the derived 3D landmark points to generate the derived 3D object model; and
storing the derived 3D object model in the computer memory.
12. The machine-readable storage medium according to claim 11 , wherein the 3D landmark algorithm computes each of the derived 3D landmark points by finding a 3D point along a ray extending from the camera center through a corresponding one of the plurality of 2D landmark points that minimizes error between the 3D point and a corresponding 3D landmark point on the existing 3D object model.
13. The machine-readable storage medium according to claim 11 , wherein:
said receiving a plurality of 2D landmark points includes receiving at least six 2D facial landmark points, wherein at least four 2D facial landmark points of the at least six 2D facial landmark points are non-colinear; and
said warping the existing 3D object model includes warping the existing 3D object model using a thin plate spline warping function to move points on the existing 3D object model other than the 3D landmark points corresponding to the derived 3D landmark points.
14. The machine-readable storage medium according to claim 11 , further comprising regularizing the derived 3D object model by executing a regularization algorithm configured to perform data completion technique.
15. The machine-readable storage medium according to claim 11 , wherein the existing 3D object model comprises a generic model of the object.
16. The machine-readable storage medium according to claim 11 , wherein the object comprises a human head, the portion of the object comprises a human face, the derived 3D object model comprises a 3D craniofacial model, the plurality of 2D landmarks comprises a plurality of 2D facial landmarks, and the existing 3D object model comprises an existing 3D craniofacial model.
17. The machine-readable storage medium according to claim 16 , wherein the 3D landmark algorithm computes each of the derived 3D landmark points by finding a 3D point along a ray extending from the camera center through a corresponding one of the plurality of 2D facial landmark points that minimizes error between the 3D point and a corresponding 3D landmark point on the existing 3D craniofacial model.
18. The machine-readable storage medium according to claim 16 , wherein:
said receiving a plurality of 2D landmark points includes receiving at least six 2D facial landmark points, wherein at least four 2D facial landmark points of the at least six 2D facial landmark points are non-colinear; and
said warping the existing 3D craniofacial model includes warping the existing 3D craniofacial model using a thin plate spline warping function to move points on the existing 3D craniofacial model other than the 3D landmark points corresponding to the derived 3D landmark points.
19. The machine-readable storage medium according to claim 16 , further comprising regularizing the derived 3D craniofacial model by executing a regularization algorithm configured to perform data completion technique.
20. The machine-readable storage medium according to claim 11 , further comprising executing a camera projection matrix algorithm that minimizes an average reprojection error.