System and method for automated simulation of teeth transformation
Provided is a system and a method for image synthesis of dental anatomy transformation. In an aspect, there is provided a method comprising: receiving an input segmentation map; transforming the input segmentation map into an input latent vector using a trained encoder; transforming the input latent vector to an output latent vector using a trained transformer machine learning model; transforming the output latent vector to an output segmentation map using a trained decoder; and outputting the output segmentation map.
1. A method for automated simulation of teeth transformation in a subject, the method comprising:
generating an input segmentation map by analyzing a plurality of pixels of an input image using a trained segmentation model, wherein at least a subset of pixels, of the plurality of pixels, corresponds to one or more exposed teeth in the subject's intraoral region;
accessing the input segmentation map, the input segmentation map comprising a semantic representation of the subject's intraoral region, wherein in the semantic representation, each tooth in the intraoral region is represented by a respective subset of pixels, of the plurality of pixels, and associated with a respective categorical label;
encoding, using a trained encoder model, the input segmentation map to generate an input latent vector, wherein the input latent vector is a compressed representation of the input segmentation map;
transforming, using a trained vector transform model, the input latent vector into an output latent vector, wherein the output latent vector is a compressed representation of an output segmentation map;
decoding, using a trained decoder, the output latent vector to generate the output segmentation map, the output segmentation map comprising a transformed semantic representation comprising, for one or more teeth, one or more respective transformed groups of pixels; and
generating an output image using the output segmentation map,
wherein transforming the input latent vector into the output latent vector further comprises the trained vector transform model inserting a representation of one or more simulated teeth in the output latent vector.
2. The method of claim 1 , wherein the one or more teeth, expressed by respective transformed group of pixels, have one or more of a transformed position, transformed orientation or transformed shape.
3. The method of claim 1 , wherein,
a segment of the plurality of pixels, in the input image, corresponds to a region outside of the subject's intraoral region, and
analyzing the plurality of pixels to generate the input segmentation map further comprises assigning the segment of pixels to an irrelevancy mask, and
wherein the segment of pixels assigned to the irrelevancy mask are excluded from the encoding, transforming and decoding and are included in the output segmentation map to define the location and shape of the intraoral region.
4. The method of claim 1 , wherein the trained encoder and trained decoder comprise an auto-encoder.
5. The method of claim 1 , further comprising training an encoder and a decoder to generate the trained encoder and the trained decoder by:
receiving a first segmentation map;
encoding, using the encoder, the first segmentation map to generate a latent vector;
decoding, using the decoder, the latent vector to generate a second segmentation map;
determining a loss function associated with a difference between the first and second segmentation maps; and
using the loss function to train an autoencoder.
6. The method of claim 1 , wherein each of the trained encoder and trained decoder use a multi-layer perceptron architecture, and wherein the multi-layer-perceptron architecture for the trained encoder can comprise a plurality of expand then squeeze layers each followed by a LeakyRelu activation except for a last layer that is followed by a TanH function to accommodate for a desired type of latent vector.
7. The method of claim 1 , wherein the trained encoder and trained decoder use a convolution neural network (CNN), wherein for the trained encoder, the activation functions comprise LeakyRelu activation except for a last convolution layer that is followed by a TanH function.
8. The method of claim 1 , wherein,
the trained encoder converts the input segmentation map of dimensions L×M×S into the input latent vector of dimensions 1×1×P, wherein “P” is a dimension count of the input latent vector, “L” and “M” are the dimensions of the input image, and S is the number of semantic categories, and
wherein the input latent vector defines an input tensor, and at one or more layers of the CNN, a resolution of the input tensor is reduced by a factor of 2×2 while concurrently doubling a number of kernels up to “P” kernels to generate the input latent vector of dimensions 1×1×P, and
the trained decoder has an inverse structure to the trained encoder and converts the output latent vector having dimensions of 1×1×P to the output segmentation map having dimensions of L×M×S.
9. A system for automated simulation of teeth transformation in a subject, the system comprising:
a memory unit for storing an input segmentation map, the input segmentation map comprising a semantic representation of the subject's intraoral region, wherein in the semantic representation, each tooth in the intraoral region is represented by a group of pixels and associated with a respective categorical label;
a computer processor coupled to the memory unit and being operable to perform a method comprising:
generating an input segmentation map by analyzing a plurality of pixels of an input image using a trained segmentation model, wherein at least a subset of pixels, of the plurality of pixels, corresponds to one or more exposed teeth in the subject's intraoral region;
encoding, using a trained encoder model, the input segmentation map to generate an input latent vector, wherein the input latent vector is a compressed representation of the input segmentation map;
transforming, using a trained vector transform model, the input latent vector into an output latent vector, wherein the output latent vector is a compressed representation of an output segmentation map;
decoding, using a trained decoder, the output latent vector to generate the output segmentation map, the output segmentation map comprising a transformed semantic representation comprising, for one or more teeth, one or more respective transformed groups of pixels; and
generating an output image using the output segmentation map,
wherein transforming the input latent vector into the output latent vector further comprises the trained vector transform model inserting a representation of one or more simulated teeth in the output latent vector.
10. The system of claim 9 , wherein the one or more teeth, expressed by respective transformed group of pixels, have one or more of a transformed position, transformed orientation or transformed shape.
11. The system of claim 9 , wherein,
a segment of the plurality of pixels, in the input image, corresponds to a region outside of the subject's intraoral region, and
analyzing the plurality of pixels to generate the input segmentation map further comprises assigning the segment of pixels to an irrelevancy mask, and
wherein the segment of pixels assigned to the irrelevancy mask are excluded from the encoding, transforming and decoding and are included in the output segmentation map to define the location and shape of the intraoral region.
12. The system of claim 9 , wherein the trained encoder and trained decoder comprise an auto-encoder.
13. The system of claim 9 , the computer processor being further operable to perform the method comprising training an encoder and a decoder to generate the trained encoder and the trained decoder by:
receiving a first segmentation map;
encoding, using the encoder, the first segmentation map to generate a latent vector;
decoding, using the decoder, the latent vector to generate a second segmentation map;
determining a loss function associated with a difference between the first and second segmentation maps; and
using the loss function to train an autoencoder.
14. The system of claim 9 , wherein each of the trained encoder and trained decoder use a multi-layer perceptron architecture, and wherein the multi-layer-perceptron architecture for the trained encoder can comprise a plurality of expand then squeeze layers each followed by a LeakyRelu activation except for a last layer that is followed by a TanH function to accommodate for a desired type of latent vector.
15. The system of claim 9 , wherein the trained encoder and trained decoder use a convolution neural network (CNN), wherein for the trained encoder, the activation functions comprise LeakyRelu activation except for a last convolution layer that is followed by a TanH function.
16. The system of claim 9 , wherein,
the trained encoder converts the input segmentation map of dimensions L×M×S into the input latent vector of dimensions 1×1×P, wherein “P” is a dimension count of the input latent vector, “L” and “M” are the dimensions of the input image, and S is the number of semantic categories, and
wherein the input latent vector defines an input tensor, and at one or more layers of the CNN, a resolution of the input tensor is reduced by a factor of 2×2 while concurrently doubling a number of kernels up to “P” kernels to generate the input latent vector of dimensions 1×1×P, and
the trained decoder has an inverse structure to the trained encoder and converts the output latent vector having dimensions of 1×1×P to the output segmentation map having dimensions of L×M×S.
17. The system of claim 9 , wherein transforming the input latent vector into the output latent vector further comprises the trained vector transform model inserting a representation of one or more simulated teeth in the output latent vector.
18. A non-transitory computer-readable medium storing computer program instructions which, when executed by a computer processor, cause the computer processor to perform a method for automated simulation of teeth transformation in a subject, the method comprising:
generating an input segmentation map by analyzing a plurality of pixels of an input image using a trained segmentation model, wherein at least a subset of pixels, of the plurality of pixels, corresponds to one or more exposed teeth in the subject's intraoral region;
accessing the input segmentation map, the input segmentation map comprising a semantic representation of the subject's intraoral region, wherein in the semantic representation, each tooth in the intraoral region is represented by a respective subset of pixels, of the plurality of pixels, and associated with a respective categorical label;
encoding, using a trained encoder model, the input segmentation map to generate an input latent vector, wherein the input latent vector is a compressed representation of the input segmentation map;
transforming, using a trained vector transform model, the input latent vector into an output latent vector, wherein the output latent vector is a compressed representation of an output segmentation map;
decoding, using a trained decoder, the output latent vector to generate the output segmentation map, the output segmentation map comprising a transformed semantic representation comprising, for one or more teeth, one or more respective transformed groups of pixels; and
generating an output image using the output segmentation map,
wherein transforming the input latent vector into the output latent vector further comprises the trained vector transform model inserting a representation of one or more simulated teeth in the output latent vector.