Generative adversarial network for dental image super-resolution, image sharpening, and denoising
A novel GAN is trained to predict high fidelity synthetic images based on low quality input dental images. The GAN further takes input anatomic masks as inputs with each image, the masks labeling pixels of the image corresponding to dental features. The GAN includes an encoder-decoder generator with semantically aware normalization between stages of the decoder according to the masks. The predicted synthetic dental image and an unpaired dental image are evaluated by a first discriminator of the GAN to obtain a realism estimate. The synthetic image and an unpaired dental image may be processed using a pretrained dental encoder to obtain a perceptual loss. The GAN is trained with the realism estimate, perceptual loss, and L1 loss. Utilization may include inputting noisy, low contrast, low resolution, blurry, or degraded dental images and outputting high resolution, denoised, high contrast, deobfuscated, and sharp dental images.
1. A method comprising:
providing, on a computer system, a plurality of training data entries, each training data entry of the plurality of training data entries including a training image and one or more training masks, each training mask of the one or more training masks indicating pixels of the training image corresponding to a dental feature of a plurality of dental features associated with the each training mask; and
for each training data entry of the plurality of training data entries, training, by the computer system, a generative adversarial network (GAN) by:
generating a training synthetic image using a generator of the GAN by processing the training image and the one or more training masks of the each training data entry;
processing the training synthetic image and a first unpaired image from a repository with a discriminator of the GAN, to obtain a realism estimate; and
updating the GAN according to the realism estimate.
2. The method of claim 1 , wherein training the generative adversarial network (GAN) further comprises, for each training data entry of the plurality of training data entries:
obtaining a level two (L2) loss by comparing the training synthetic image to a paired image, the training image being derived from the paired image; and
wherein updating the GAN according to the realism estimate comprises updating the GAN according to both of the realism estimate and the L2 loss.
3. The method of claim 2 , wherein the training image is a degraded version of the paired image.
4. The method of claim 3 , wherein the training image is obtained from the paired image by any of distorting, blurring, and adding noise to the paired image.
5. The method of claim 2 , wherein the training image is a lower resolution version of the paired image.
6. The method of claim 2 , wherein training the generative adversarial network (GAN) further comprises, for each training data entry of the plurality of training data entries:
processing the training synthetic image with a pretrained encoder to obtain first intermediate values from one or more stages of the pretrained encoder, the pretrained encoder being trained to identify an item of dental anatomy or an item of dental restoration;
processing a second unpaired image with the pretrained encoder to obtain second intermediate values from one or more stages of the pretrained encoder, the second unpaired image being either the first unpaired image or a different unpaired image; and
obtaining a perceptual loss according to differences between the first intermediate values and the second intermediate values;
wherein updating the GAN according to the realism estimate and the L2 loss further comprises updating the GAN according to all of the realism estimate, L2 loss, and the perceptual loss.
7. The method of claim 1 , wherein the training synthetic image has a first resolution at least twice a second resolution of the training image.
8. The method of claim 7 , wherein the generator includes an encoder and a decoder, an input stage of the encoder taking as input an input matrix having two first dimensions and the decoder produces an output matrix having two second dimensions that are larger than the two first dimensions.
9. The method of claim 8 , wherein the decoder has more stages than the encoder.
10. The method of claim 1 , wherein the plurality of dental features include a plurality of types of dental anatomy and a plurality of types of dental treatments.
11. A computer-readable medium that is non-transitory and stores executable code, that when executed by one or more processors, causes the one or more processors to perform a first method comprising:
receiving a plurality of training data entries, each training data entry of the plurality of training data entries including a training image and one or more training masks, each training mask of the one or more training masks indicating pixels of the training image corresponding to a dental feature of a plurality of dental features associated with the each training mask; and
for each training data entry of the plurality of training data entries, training a generative adversarial network (GAN) by:
generating a training synthetic image using a generator of the GAN by processing the training image and the one or more training masks of the each training data entry;
processing the training synthetic image and a first unpaired image from a repository with a discriminator of the GAN, to obtain a realism estimate; and
updating the GAN according to the realism estimate.
12. The computer-readable medium of claim 11 , wherein training the generative adversarial network (GAN) further comprises, for each training data entry of the plurality of training data entries:
obtaining a level two (L2) loss by comparing the training synthetic image to a paired image, the training image being derived from the paired image; and
wherein updating the GAN according to the realism estimate comprises updating the GAN according to both of the realism estimate and the L2 loss.
13. The computer-readable medium of claim 12 , wherein the training image is a degraded version of the paired image.
14. The computer-readable medium of claim 13 , wherein the training image is obtained from the paired image by any of distorting, blurring, and adding noise to the paired image.
15. The computer-readable medium of claim 12 , wherein the training image is a lower resolution version of the paired image.
16. The computer-readable medium of claim 12 , wherein training the generative adversarial network (GAN) further comprises, for each training data entry of the plurality of training data entries:
processing the training synthetic image with a pretrained encoder to obtain first intermediate values from one or more stages of the pretrained encoder, the pretrained encoder being trained to identify an item of dental anatomy or an item of dental restoration;
processing a second unpaired image with the pretrained encoder to obtain second intermediate values from one or more stages of the pretrained encoder, the second unpaired image being either the first unpaired image or a different unpaired image; and
obtaining a perceptual loss according to differences between the first intermediate values and the second intermediate values;
wherein updating the GAN according to the realism estimate and the L2 loss further comprises updating the GAN according to all of the realism estimate, L2 loss, and the perceptual loss.
17. The computer-readable medium of claim 11 , wherein the training synthetic image has a first resolution at least twice a second resolution of the training image.
18. The computer-readable medium of claim 17 , wherein the generator includes an encoder and a decoder, an input stage of the encoder taking as input an input matrix having two first dimensions and the decoder produces an output matrix having two second dimensions that are larger than the two first dimensions.
19. The computer-readable medium of claim 18 , wherein the decoder has more stages than the encoder.
20. The computer-readable medium of claim 11 , wherein the plurality of dental features include a plurality of types of dental anatomy and a plurality of types of dental treatments.