Photograph-based assessment of dental treatments and procedures
The current document is directed to methods and systems for monitoring a dental patient's progress during a course of treatment. A three-dimensional model of the expected positions of the patient's teeth can be projected, in time, from a three-dimensional model of the patient's teeth prepared prior to beginning the treatment. A digital camera is used to take one or more two-dimensional photographs of the patient's teeth, which are input to a monitoring system. The monitoring system determines virtual-camera parameters for each two-dimensional input image with respect to the time-projected three-dimensional model, uses the determined virtual-camera parameters to generate two-dimensional images from the three-dimensional model, and then compares each input photograph to the corresponding generated two-dimensional image in order to determine how closely the three-dimensional arrangement of the patient's teeth corresponds to the time-projected three-dimensional arrangement.
1. A dental-treatment monitoring system comprising a computing device with a non-transitory computer-readable data-storage having instructions which can be executed by one or more processors to cause the computing device to:
receive two-dimensional image data of a patient's teeth taken at a particular time during a course of dental treatment, the two-dimensional image data including associated metadata; and
using the one or more processors:
compare the two-dimensional image data with a three-dimensional model representing an expected configuration of the patient's teeth at the particular time;
determine a set of virtual-camera parameters that describe an estimated position and orientation of a virtual camera that produces a generated image from the three-dimensional model;
determine an iteration variable N based on the metadata associated with the two-dimensional image data;
iteratively modify the set of virtual-camera parameters N times to modify the generated image, wherein iterative modification includes:
a first adjustment of a current set of virtual-camera parameters for the generated image;
a second adjustment of the current set of virtual-camera parameters for the generated image, wherein the second adjustment includes: for each of two jaws of the patient, masking out the other of the two jaws and associated teeth from the generated image, and optimizing the set of virtual-camera parameters for the generated image with respect to a pixel-associated cost for the generated image and a next corresponding generated image;
a first refinement of the current set of virtual-camera parameters for the generated image; and
a second refinement of the current set of virtual-camera parameters for the generated image; and
generate a comparison value for the two-dimensional image compared to the three-dimensional model.
2. The system of claim 1 , wherein the pixel-associated cost for the generated image and the next corresponding generated image is computed as a linear combination of mutual information for the generated image and the next corresponding generated image and a sum of pixel-associated costs.
3. The system of claim 1 , wherein optimizing the set of virtual-camera parameters for the generated image comprises: for each optimization iteration:
generating the next corresponding generated image from the three-dimensional model of the patient's teeth using the current set of virtual-camera parameters for the generated image;
computing a cost for the generated image and the next corresponding generated image; and
perturbing the set of virtual-camera parameters for the generated image in a direction that minimizes the cost for the generated image and the next corresponding generated image.
4. The system of claim 1 , wherein optimizing the set of virtual-camera parameters for the generated image comprises application of a Nelder-Mead downhill simplex optimization method, with seven dimensions, including three rotations, three translations, and a virtual-camera view angle.
5. The system of claim 1 , wherein the first refinement of the current set of virtual-camera parameters for the generated image comprises: for each tooth group of a currently considered jaw:
masking the generated image to leave the tooth group of the currently considered jaw, and
optimizing the set of virtual-camera parameters for the generated image with respect to a gradient-vector-based cost for the generated image and the next corresponding generated image.
6. The system of claim 1 , wherein the first refinement of the current set of virtual-camera parameters for the generated image comprises: for each tooth group of a currently considered jaw:
masking the generated image to leave the tooth group of the currently considered jaw, and
optimizing the set of virtual-camera parameters for the generated image with respect to a Laplacian-vector-based cost for the generated image and the next corresponding generated image.
7. The system of claim 1 , wherein the metadata include one or more of: a text label, an indication of a standard type of view represented by the two-dimensional image, an image size, a date and time information, an indication of a camera model and make, an indication of a camera orientation, an indication of an aperture, an indication of a shutter speed, an indication of a focal length, an indication of a metering mode, and an indication of an image speed.
8. The system of claim 1 , wherein the instructions further cause the computing device to determine, from the comparison value, whether a configuration of the patient's teeth is within a threshold level of correspondence to the expected configuration of the patient's teeth.
9. The system of claim 1 , wherein the instructions further cause the computing device to remove teeth from the two-dimensional image that are not considered for producing the generated image.
10. The system of claim 1 , wherein iteratively modifying the set of virtual-camera parameters comprises:
applying a color and intensity threshold value to each pixel of the generated image;
determining whether each pixel corresponds to a tooth region or a non-tooth region based on the color and intensity threshold value; and
generating a teeth mask for the generated image.
11. The system of claim 1 , wherein the instructions further cause the computing device to generate multiple comparison values for multiple two-dimensional images.
12. The system of claim 1 , wherein iteratively modifying the set of virtual-camera parameters further comprises:
thresholding the generated image to generate a first teeth mask;
thresholding a next corresponding image generated from the three-dimensional model of the patient's teeth using the current set of virtual-camera parameters for the generated image to generate a next second corresponding teeth mask,
generating a distance transform of the first teeth mask,
searching over scalings, rotations, and translations of the next second corresponding teeth mask to identify a next minimum-cost overlay of the next second corresponding teeth mask over the distance transform of the first teeth mask, and
adjusting the set of virtual-camera parameters for the generated image corresponding to the next minimum-cost overlay; and
selecting, as an adjusted set of virtual-camera parameters for the generated image, the set of virtual-camera parameters associated with lowest-cost overlay of any of the generated next second corresponding teeth mask over the distance transform of the first teeth mask.
13. The system of claim 12 , wherein a cost of an overlay of the next second corresponding teeth mask over the distance transform of the first teeth mask is a sum of distances associated with pixels in the distance transform of the first teeth mask underlying a tooth region within the next second corresponding teeth mask.
14. The system of claim 12 , wherein adjusting the set of virtual-camera parameters for the generated image corresponding to the next minimum-cost overlay includes:
adjusting the set of virtual-camera parameters to move a position of a center of the next corresponding image generated from a time-projected three-dimensional model of the patient's teeth;
adjusting the set of virtual-camera parameters to rotate the next corresponding image generated;
from the time-projected three-dimensional model of the patient's teeth in an image plane;
adjusting the set of virtual-camera parameters to rotate the next corresponding image generated from the time-projected three-dimensional model of the patient's teeth about an axis parallel to the image plane; and
adjusting the set of virtual-camera parameters to rescale the next corresponding image generated from the time-projected three-dimensional model of the patient's teeth.
15. A dental-treatment monitoring system comprising a computing device with a non-transitory computer-readable data-storage having instructions which can be executed by one or more processors to cause the computing device to:
receive two-dimensional image data of a patient's teeth taken at a particular time during a course of dental treatment, the two-dimensional image data including associated metadata; and
using the one or more processors:
compare the two-dimensional image with a three-dimensional model representing an expected configuration of the patient's teeth at the particular time; determine a set of virtual-camera parameters that describe an estimated position and orientation of a virtual camera that produces a generated image from the three-dimensional model;
determine an iteration variable N based on the metadata associated with the two-dimensional image data;
iteratively modify the set of virtual-camera parameters N times to modify the generated image, wherein iterative modification includes:
a first adjustment of a current set of virtual-camera parameters for the generated image;
a second adjustment of the current set of virtual-camera parameters for the generated image;
a first refinement of the current set of virtual-camera parameters for the generated image, the first refinement including, for each tooth group of a currently considered jaw: masking the generated image to leave the tooth group of the currently considered jaw, and optimizing the set of virtual-camera parameters for the generated image with respect to a gradient-vector-based cost for the generated image and a next corresponding generated image; and
a second refinement of the current set of virtual-camera parameters for the generated image; and
generate a comparison value for the two-dimensional image compared to the three-dimensional model.
16. A dental-treatment monitoring system comprising a computing device with a non-transitory computer-readable data-storage having instructions which can be executed by one or more processors to cause the computing device to:
receive two-dimensional image data of a patient's teeth taken at a particular time during a course of dental treatment, the two-dimensional image data including associated metadata; and
using the one or more processors:
compare the two-dimensional image with a three-dimensional model representing an expected configuration of the patient's teeth at the particular time;
determine a set of virtual-camera parameters that describe an estimated position and orientation of a virtual camera that produces a generated image from the three-dimensional model;
determine an iteration variable N based on the metadata associated with the two-dimensional image data;
iteratively modify the set of virtual-camera parameters N times to modify the generated image, wherein iterative modification includes:
a first adjustment of a current set of virtual-camera parameters for the generated image;
a second adjustment of the current set of virtual-camera parameters for the generated image, the first adjustment including, for each tooth group of a currently considered jaw: masking the generated image to leave the tooth group of the currently considered jaw, and optimizing the set of virtual-camera parameters for the generated image with respect to a Laplacian-vector-based cost for the generated image and a next corresponding generated image;
a first refinement of the current set of virtual-camera parameters for the generated image; and
a second refinement of the current set of virtual-camera parameters for the generated image; and
generate a comparison value for the two-dimensional image compared to the three-dimensional model.