2D-to-3D tooth reconstruction, optimization, and positioning frameworks using a differentiable renderer
Provided herein are systems and methods for optimizing a 3D model of an individual's teeth. A 3D dental model may be reconstructed from 3D parameters. A differentiable renderer may be used to derive a 2D rendering of the individual's dentition. 2D image(s) of an individual's dentition may be obtained, and features may be extracted from the 2D image(s). Image loss between the 2D rendering and the 2D image(s) can be derived, and back-propagation from the image loss can be used to calculate gradients of the loss to optimize the 3D parameters. A machine learning model can also be trained to predict a 3D dental model from 2D images of an individual's dentition.
1. A method of forming a three-dimensional (3D) dental model of an individual's dentition, the method comprising:
obtaining a plurality of original two-dimensional (2D) images as part of a digital scan of the individual's dentition, wherein each original 2D image comprises a different set of camera parameters;
obtaining a set of 3D parameters for the individual's dentition;
constructing a parametric 3D dental model of the individual's dentition with the set of 3D parameters;
applying a differentiable renderer to the parametric 3D dental model to derive a plurality of 2D renderings of the individual's dentition;
extracting, for each original 2D image of the plurality of original 2D images, features from the original 2D images;
comparing the extracted features from the plurality of original 2D images to the plurality of 2D renderings and aggregating differences at each pixel location to derive an aggregate image loss function from a difference of the original 2D images and the 2D rendering at each pixel location;
performing back-propagation from the aggregate image loss function to calculate gradients of loss with respect to the set of 3D parameters;
updating the set of 3D parameters based on the calculated gradients;
revising the parametric 3D dental model of the individual's dentition with the updated set of 3D parameters; and
outputting the parametric 3D dental model.
2. The method of claim 1 , further comprising repeating for each original 2D image of the plurality of original 2D images, until the 2D renderings each converges with the extracted features within a predetermined threshold: applying the differentiable renderer to the parametric 3D dental model to derive the 2D rendering of the individual's dentition, obtaining the original 2D image of the plurality of original 2D images of the individual's dentition; extracting features from the original 2D image of the plurality of original 2D images; comparing the 2D rendering to the extracted features to derive the aggregate image loss function from the difference of the original 2D image of the plurality of original 2D images and the 2D rendering at each pixel location; performing back-propagation from the aggregate image loss function to calculate gradients of loss with respect to the set of 3D parameters; updating the set of 3D parameters based on the calculated gradients; and revising the parametric 3D dental model of the individual's dentition with the updated set of 3D parameters.
3. The method of claim 1 , wherein the parametric 3D dental model comprises a tooth mesh.
4. The method of claim 1 , wherein the parametric 3D dental model comprises a tooth point cloud.
5. The method of claim 1 , wherein the parametric 3D dental model comprises a reduced representation of a 3D dental model.
6. The method of claim 1 , wherein the parametric 3D dental model comprises a principal component analysis representation.
7. The method of claim 1 , wherein the extracted features comprise tooth masks for an upper jaw, a lower or an upper and lower jaw.
8. The method of claim 1 , wherein the extracted features comprise tooth segmentation data and/or numbering data.
9. The method of claim 1 , wherein the extracted features comprise dental edge information.
10. The method of claim 1 , further comprising:
comparing the parametric 3D dental model to an orthodontic treatment plan of the individual; and
determining whether the individual's dentition is tracking the orthodontic treatment plan.
11. A system for forming a three-dimensional (3D) dental model of an individual's dentition, the system comprising:
one or more processors; and
a memory coupled to the one or more processors, the memory storing computer-program instructions, that, when executed by the one or more processors, perform a computer-implemented method comprising:
obtaining a plurality of two-dimensional (2D) images as part of a digital scan of the individual's dentition, wherein each 2D image comprises a different set of camera parameters;
obtaining a set of 3D parameters for the individual's dentition;
constructing a parametric 3D dental model of the individual's dentition with the set of 3D parameters;
applying a differentiable renderer to the parametric 3D dental model to derive a plurality of 2D rendering of the individual's dentition;
extracting, for each 2D image of the plurality of 2D images, features from a plurality of original 2D images;
comparing the extracted features from each 2D imade of the plurality of 2D images to the plurality of 2D renderings and aggregating differences at each pixel location to derive an aggregate image loss function from a difference of the plurality of original 2D images and the 2D rendering at each pixel location;
performing back-propagation from the aggregate image loss function to calculate gradients of loss with respect to the set of 3D parameters;
updating the set of 3D parameters based on the calculated gradients;
revising the parametric 3D dental model of the individual's dentition with the updated set of 3D parameters; and
outputting the parametric 3D dental model.
12. The system of claim 11 , wherein the computer-implemented method is further configured to repeat for each of the plurality of original 2D images, until the 2D renderings each converges with the extracted features within a predetermined threshold: applying the differentiable renderer to the parametric 3D dental model to derive the 2D rendering of the individual's dentition, obtaining the original 2D image of the individual's dentition; extracting features from the original 2D image of the plurality of original 2D images; comparing the 2D rendering to the extracted features to derive the aggregate image loss function from the difference of the original 2D image of the plurality of original 2D images and the 2D rendering at each pixel location; performing back-propagation from the aggregate image loss function to calculate gradients of loss with respect to the set of 3D parameters; updating the set of 3D parameters based on the calculated gradients; and revising the parametric 3D dental model of the individual's dentition with the updated set of 3D parameters.
13. The system of claim 11 , wherein constructing the parametric 3D dental model of the individual's dentition comprises constructing a tooth mesh.
14. The system of claim 11 , wherein constructing the parametric 3D dental model of the individual's dentition comprises constructing a tooth point cloud.
15. The system of claim 11 , wherein constructing the parametric 3D dental model of the individual's dentition comprises constructing a reduced representation of a 3D dental model.
16. The system of claim 11 , wherein constructing the parametric 3D dental model of the individual's dentition comprises constructing a principal component analysis representation.
17. The system of claim 11 , wherein extracting features from the original 2D image comprises extracting tooth masks for an upper jaw, a lower or an upper and lower jaw.
18. The system of claim 11 wherein extracting features from the original 2D image comprises extracting tooth segmentation data and/or numbering data.
19. The system of claim 11 , wherein extracting features from the original 2D image comprises extracting dental edge information.
20. The system of claim 11 , wherein the computer implemented method further comprises:
comparing the parametric 3D dental model to an orthodontic treatment plan of the individual; and
determining whether the individual's dentition is tracking the orthodontic treatment plan.