Automatic marker-less alignment of digital 3D face and jaw models
The invention aligns a digital face model from a 3D face scanner with a 3D jaw scan from an intraoral scan produced by an intraoral scanner without using external markers. The alignment proceeds in two steps. In the first step, the teeth part of a subject whose teeth are clenched, referred to as a clenched-teeth face scan, is aligned with an intraoral jaw scan to obtain a first transformation matrix. In the second step, a face model of a subject with a normal facial expression is aligned with the clenched-teeth face model of the same subject to obtain a second transformation matrix. A graphical user interface is provided that enables a user to manually align the 3D jaw scan with a 2D image of the subject's teeth to determine the first transformation.
1. A computer-implemented method for aligning a face scan with a jaw scan, comprising:
receiving (1) a frontal 2D image of the face of a subject, (2) a clenched-teeth face scan from a 3D scanner, wherein a face scan is a 3D model of the face of the subject and wherein the subject's teeth are visible in the clenched-teeth face scan, and (3) a normal face scan from the 3D scanner;
receiving a jaw scan from an intraoral scanner wherein a jaw scan is a 3D model that represents the jaw and teeth of the subject;
providing a graphical user interface to a user that enables the user to manually align a visual representation of the jaw scan with respect to the frontal 2D image;
receiving from the graphical user interface alignment parameters that specify the manual alignments performed by the user;
determining a first transformation that aligns the clenched-teeth face scan with the jaw scan based on the received alignment parameters;
determining a second transformation that aligns the normal face scan with the clenched-teeth face scan, wherein said second transformation specifies a rotation and a translation in 3 dimensions;
computing weights for feature points detected in the clenched-teeth face model, wherein higher valued weights indicate features that are of greater importance and lower valued weights indicate features that are of lesser importance; and
applying the first and second transformation to the normal face scan to align it with the jaw scan.
2. The method of claim 1 , further comprising: applying the first transformation to the clenched-teeth face scan to align it with the jaw scan.
3. The method of claim 1 , wherein said determining a first transformation does not require the use of an external marker.
4. The method of claim 1 , wherein manually aligning a visual representation of the jaw scan comprises:
translating, in 3 dimensions, the jaw scan with respect to the frontal 2D mage; and rotating, in 3 dimensions, the jaw scan with respect to the frontal 2D image.
5. The method of claim 4 , wherein said alignment parameters specify a jaw to teeth rigid body transformation based on the translation and rotation performed by the user using the graphical user interface.
6. The method of claim 5 , wherein said first transformation is the inverse of the jaw to teeth rigid body transformation.
7. The method of claim 1 , wherein said clenched-teeth face scan depicts a subject whose top row and bottom row of teeth are both touching and visible.
8. The method of claim 1 , wherein said jaw scan comprises either a 3D model of the subject's upper and lower jaw or a 3D model of only the upper jaw.
9. The method of claim 1 , wherein said determining a second transformation does not require the use of an external marker.
10. The method of claim 1 , wherein said normal face scan depicts a subject that is smiling.
11. The method of claim 1 , wherein said determining a second transformation comprises:
detecting feature points in the normal face model;
detecting feature points in the clenched-teeth face
model; computing an initial transformation; and
determining the second transformation based on the initial transformation.
12. The method of claim 1 , wherein determining the second transformation uses the iterative closest point (ICP) algorithm.
13. A device for aligning a face scan with a jaw scan, comprising a processor;
a memory in communication with the processor for storing instructions, which when executed by the processor cause the device:
to receive (1) a frontal 2D image of the face of a subject, and (2) a clenched-teeth face scan from a 3D scanner wherein a face scan is a 3D model of the face of a subject and wherein the subject's teeth are visible in the clenched-teeth face scan, and (3) a normal face scan from the 3D scanner;
to receive a jaw scan from an intraoral scanner wherein a jaw scan is a 3D model that represents the jaw and frontal teeth of the subject;
to provide a graphical user interface to a user that enables the user to manually align a visual representation of the jaw scan with respect to the frontal 2D image;
to receive from the graphical user interface alignment parameters that specify the manual alignments performed by the user;
to determine a first transformation that aligns the clenched-teeth face scan with the jaw scan;
to determine a second transformation that aligns the normal face scan with the clenched-teeth face scan, wherein said second transformation specifies a rotation and a translation in 3 dimensions; to compute weights for feature points detected in the clenched-teeth face model, wherein higher valued weights indicate features that are of greater importance and lower valued weights indicate features that are of lesser importance; and
applying the first and second transformation to the normal face scan to align it with the jaw scan.
14. The device of claim 13 , wherein the instructions, when executed by the processor, further cause the device:
to apply the first transformation to the clenched-teeth face scan to align it with the
jaw scan.
15. The device of claim 13 , wherein said determining a first transformation does not require the use of an external marker.
16. The device of claim 13 , wherein manually aligning a visual representation of the jaw scan comprises:
translating, in 3 dimensions, the jaw scan with respect to the frontal 2D mage; and rotating, in 3 dimensions, the jaw scan with respect to the frontal 2D image.
17. The device of claim 16 , wherein said alignment parameters specify a jaw to teeth rigid body transformation based on the translation and rotation performed by the user using the graphical user interface.
18. The device of claim 17 , wherein said first transformation is the inverse of the jaw to teeth rigid body transformation.
19. The device of claim 13 , wherein said clenched-teeth face scan depicts a subject whose top row and bottom row of teeth are both touching and visible.
20. The device of claim 13 , wherein said jaw scan comprises either a 3D model of the subject's upper and lower jaw or a 3D model of only the upper jaw.
21. The device of claim 13 , wherein said determining a second transformation does not require the use of an external marker.
22. The device of claim 13 , wherein said normal face scan depicts a subject that is smiling.
23. The device of claim 13 , wherein said determining a second transformation comprises:
detecting feature points in the normal face model;
detecting feature points in the clenched-teeth face
model; computing an initial transformation; and
determining the second transformation based on the initial transformation.
24. The device of claim 13 , wherein determining the second transformation uses the iterative closest point (ICP) algorithm.