Automated 2D/3D integration and lip spline autoplacement
A method may include: receiving an image of a mouth region of a patient's face; extracting teeth contours within the image of the mouth region of the patient's face; locating a mouth opening within the image of the mouth region of the patient's face; extracting the tooth contours from a 3D model of the patient's teeth; and aligning the tooth contours from the 3D model with the tooth contours of the teeth within the image of the mouth region of the patient's face.
1. A computer-implemented method of virtually representing an orthodontic treatment outcome of a patient's teeth, the computer-implemented method comprising:
receiving an image of a mouth region of the patient's face;
extracting teeth contours within the image of the mouth region of the patient's face by:
detecting the teeth contours,
binarizing the teeth contours, and
thinning the teeth contours;
locating a mouth opening within the image of the mouth region of the patient's face;
extracting the teeth contours from a 3D model of the patient's teeth; and
aligning the teeth contours from the 3D model with the teeth contours of the patient's teeth within the image of the mouth region of the patient's face.
2. The computer-implemented method of claim 1 , further comprising:
inserting a rendering of the 3D model into the mouth opening based on the alignment of the teeth contours from the 3D model with the teeth contours of the patient's teeth within the image of the mouth region of the patient's face.
3. The computer-implemented method of claim 1 , wherein receiving the image of the mouth region of the patient's face comprises:
receiving a facial image of the patient;
identifying facial landmarks on the facial image, the facial landmarks including lip landmarks and other landmarks;
cropping the facial image around the lip landmarks to exclude the other landmarks.
4. The computer-implemented method of claim 3 , wherein the other landmarks include one or more of eye, nose, and facial outline landmarks.
5. The computer-implemented method of claim 1 , wherein detecting the teeth contours includes detecting the teeth contours using a first convolutional neural network that comprises a holistic edge detection deep learning model.
6. The computer-implemented method of claim 1 , wherein each pixel of the teeth contours has a value.
7. The computer-implemented method of claim 1 , wherein binarizing the teeth contours comprises:
comparing a value of each pixel to a threshold; and
assigning a new value to each pixel, the new value being a first value if the pixel is greater than the threshold and a second value if the pixel is less than the threshold.
8. The computer-implemented method of claim 1 , wherein thinning the teeth contours comprises reducing the width of the teeth contours to a single pixel at each location along the teeth contours.
9. The computer-implemented method of claim 1 , wherein aligning the teeth contours from the 3D model with the teeth contours of the patient's teeth within the image of the mouth region of the patient's face comprises:
using an expectation-maximization algorithm to align the teeth contours from the 3D model with the teeth contours of the patient's teeth within the image, where during an expectation-step, each pixel on the teeth contours from the 3D model is matched to a similar pixel on the teeth contours of the 3D model and during a maximization-step, the patient's teeth are adjusted in one or more of translation and rotation in one or more of three orthogonal direction to minimize the total discrepancies between pixels of the teeth contours from the 3D model and the teeth contours of the patient's teeth within the image.
10. The computer-implemented method of claim 1 , wherein locating the mouth opening within the image of the mouth region of the patient's face comprises:
detecting lip contours using a second convolutional neural network;
binarizing the lip contours; and
thinning the lip contours.
11. The computer-implemented method of claim 10 , wherein the second convolutional neural network comprises a holistic edge detection deep learning model.
12. The computer-implemented method of claim 10 , wherein each pixel of the lip contours has a value.
13. The computer-implemented method of claim 10 , wherein binarizing the lip contours comprises:
comparing a value of each pixel to a threshold; and
assigning a new value to each pixel, the new value being a first value if the pixel is greater than the threshold and a second value if the pixel is less than the threshold.
14. The computer-implemented method of claim 10 , wherein thinning the lip contours comprises reducing the width of the lip contours to a single pixel at each location along the lip contours.
15. The computer-implemented method of claim 10 , further comprising:
forming a first plurality of connected splines along a lower lip portion of the lip contours, the first plurality of connected splines starting at a first end of the lower lip portion of the lip contours and ending at a second end of the lower lip portion of the lip contours; and
forming a second plurality of connected splines along an upper lip portion of the lip contours, the second plurality of connected splines starting at a first end of the upper lip portion of the lip contours and ending at a second end of the upper lip portion of the lip contours, wherein the first and second plurality of connected splines define the mouth opening.
16. A system comprising:
memory storing computer-program instructions; and
one or more physical processors coupled to the memory, the one or more physical processors configured to implement a computer-implemented method using the computer-program instructions, the computer-implemented method of virtually representing an orthodontic treatment outcome using automated detection of facial and dental reference objects, the computer-implemented method comprising:
receiving an image of a mouth region of a patient's face;
extracting teeth contours within the image of the mouth region of the patient's face by:
detecting the teeth contours using a first convolutional neural network,
binarizing the teeth contours, and
thinning the teeth contours;
locating a mouth opening within the image of the mouth region of the patient's face;
extracting the teeth contours from a 3D model of the patient's teeth; and
aligning the teeth contours from the 3D model with the teeth contours of the patient's teeth within the image of the mouth region of the patient's face.
17. The system of claim 16 , further comprising:
inserting a rendering of the 3D model into the mouth opening based on the alignment of the teeth contours from the 3D model with the teeth contours of the patient's teeth within the image of the mouth region of the patient's face.
18. The system of claim 16 , wherein receiving the image of the mouth region of the patient's face comprises:
receiving a facial image of the patient;
identifying facial landmarks on the facial image, the facial landmarks including lip landmarks and other landmarks;
cropping the facial image around the lip landmarks to exclude the other landmarks.
19. The system of claim 18 , wherein the other landmarks include one or more of eye, nose, and facial outline landmarks.
20. The system of claim 16 , wherein the first convolutional neural network comprises a holistic edge detection deep learning model.
21. The system of claim 20 , wherein each pixel of the teeth contours has a value.
22. The system of claim 16 , wherein binarizing the teeth contours comprises:
comparing a value of each pixel to a threshold; and
assigning a new value to each pixel, the new value being a first value if the pixel is greater than the threshold and a second value if the pixel is less than the threshold.
23. The system of claim 16 , wherein thinning the teeth contours comprises reducing the width of the teeth contours to a single pixel at each location along the teeth contours.
24. The system of claim 16 , wherein aligning the teeth contours from the 3D model with the teeth contours of the patient's teeth within the image of the mouth region of the patient's face comprises:
using an expectation-maximization algorithm to align the teeth contours from the 3D model with the teeth contours of the patient's teeth within the image, where during an expectation-step, each pixel on the teeth contours from the 3D model is matched to a similar pixel on the teeth contours of the 3D model and during a maximization-step, the patient's teeth are adjusted in one or more of translation and rotation in one or more of three orthogonal direction to minimize the total discrepancies between pixels of the teeth contours from the 3D model and the teeth contours of the patient's teeth within the image.
25. The system of claim 16 , wherein locating the mouth opening within the image of the mouth region of the patient's face comprises:
detecting lip contours using a second convolutional neural network;
binarizing the lip contours; and
thinning the lip contours.