Correcting differences in multi-scanners for digital pathology images using deep learning
The present disclosure relates to techniques for transforming digital pathology images obtained by different slide scanners into a common format for image analysis. Particularly, aspects of the present disclosure are directed to obtaining a source image of a biological specimen, the source image is generated from a first type of scanner, inputting into a generator model a randomly generated noise vector and a latent feature vector from the source image as input data, generating, by the generator model, a new image based on the input data, inputting into a discriminator model the new image, generating, by the discriminator model, a probability for the new image being authentic or fake, determining whether the new image is authentic or fake based on the generated probability, and outputting the new image when the image is authentic.
1. A method comprising:
obtaining a source image of a biological specimen, wherein the source image is generated from a first type of scanner;
inputting into a generator model a randomly generated noise vector and a latent feature vector from the source image as input data;
generating, by the generator model, a new image based on the input data;
inputting into a discriminator model the new image;
generating, by the discriminator model, a probability for the new image being authentic or fake, wherein authentic means the new image has characteristics that are similar to characteristics of a target image, and fake means the new image does not have the characteristics that are similar to the characteristics of the target image, and wherein the characteristics of the target image are associated with a second type of scanner that is different from the first type of scanner;
determining whether the new image is authentic or fake based on the generated probability; and
outputting the new image when the image is authentic.
2. The method of claim 1 , wherein the biological specimen is mounted on a pathology slide, the first type of scanner is a first type of whole slide imaging scanner, and the second type of scanner is a second type of whole slide imaging scanner.
3. The method of claim 1 , further comprising:
inputting into an image analysis model the new image, wherein the image analysis model comprises a plurality of model parameters learned using a set of training data comprising images obtained from a same type of scanner as the second type of scanner;
analyzing, by the image analysis model, the new image;
generating, by the image analysis model, an analysis result based on the analyzing of the new image; and
outputting the analysis result.
4. The method of claim 3 , wherein the image analysis model is not trained on images obtained from a same type of scanner as the first type of scanner.
5. The method of claim 1 , further comprising training an image analysis model using a set of training data comprising the new image.
6. The method of claim 1 , wherein the generator model and the discriminator model are part of a Generative Adversarial Network (GAN) model.
7. The method of claim 6 , wherein:
the GAN model comprises a plurality of model parameters learned using a set of training data comprising one or more pairwise sets of images, wherein each pair of images within the one or more pairwise sets of images comprises a first image generated by the first type of scanner and a second image generated by the second type of scanner; and
wherein the plurality of model parameters are learned using the set of training data based on minimizing a first loss function to train the discriminator model to maximize a probability of the set of training data and a second loss function to train the discriminator model to minimize a probability of a generated image sampled from the generator model and train the generator model to maximize the probability that the discriminator model assigns to the generated image.
8. The method of claim 3 , further comprising: determining, by a user, a diagnosis of a subject based on the analysis result.
9. The method of claim 8 , further comprising administering, by the user, a treatment with a compound based on (i) the analysis result, and/or (iii) the diagnosis of the subject.
10. A system comprising:
one or more data processors; and
a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform actions including:
obtaining a source image of a biological specimen, wherein the source image is generated from a first type of scanner;
inputting into a generator model a randomly generated noise vector and a latent feature vector from the source image as input data;
generating, by the generator model, a new image based on the input data;
inputting into a discriminator model the new image;
generating, by the discriminator model, a probability for the new image being authentic or fake, wherein authentic means the new image has characteristics that are similar to characteristics of a target image, and fake means the new image does not have the characteristics that are similar to the characteristics of the target image, and wherein the characteristics of the target image are associated with a second type of scanner that is different from the first type of scanner;
determining whether the new image is authentic or fake based on the generated probability; and
outputting the new image when the image is authentic.
11. The system of claim 10 , wherein the biological specimen is mounted on a pathology slide, the first type of scanner is a first type of whole slide imaging scanner, and the second type of scanner is a second type of whole slide imaging scanner.
12. The system of claim 10 , wherein the actions further include:
inputting into an image analysis model the new image, wherein the image analysis model comprises a plurality of model parameters learned using a set of training data comprising images obtained from a same type of scanner as the second type of scanner;
analyzing, by the image analysis model, the new image;
generating, by the image analysis model, an analysis result based on the analyzing of the new image; and
outputting the analysis result.
13. The system of claim 12 , wherein the image analysis model is not trained on images obtained from a same type of scanner as the first type of scanner.
14. The system of claim 13 , wherein the actions further include training an image analysis model using a set of training data comprising the new image.
15. The system of claim 10 , wherein:
the generator model and the discriminator model are part of a Generative Adversarial Network (GAN) model:
the GAN model comprises a plurality of model parameters learned using a set of training data comprising one or more pairwise sets of images, wherein each pair of images within the one or more pairwise sets of images comprises a first image generated by the first type of scanner and a second image generated by the second type of scanner; and
wherein the plurality of model parameters are learned using the set of training data based on minimizing a first loss function to train the discriminator model to maximize a probability of the set of training data and a second loss function to train the discriminator model to minimize a probability of a generated image sampled from the generator model and train the generator model to maximize the probability that the discriminator model assigns to the generated image.
16. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform actions including:
obtaining a source image of a biological specimen, wherein the source image is generated from a first type of scanner;
inputting into a generator model a randomly generated noise vector and a latent feature vector from the source image as input data;
generating, by the generator model, a new image based on the input data;
inputting into a discriminator model the new image;
generating, by the discriminator model, a probability for the new image being authentic or fake, wherein authentic means the new image has characteristics that are similar to characteristics of a target image, and fake means the new image does not have the characteristics that are similar to the characteristics of the target image, and wherein the characteristics of the target image are associated with a second type of scanner that is different from the first type of scanner;
determining whether the new image is authentic or fake based on the generated probability; and
outputting the new image when the image is authentic.
17. The computer-program product of claim 16 , wherein the biological specimen is mounted on a pathology slide, the first type of scanner is a first type of whole slide imaging scanner, and the second type of scanner is a second type of whole slide imaging scanner.
18. The computer-program product of claim 16 , wherein the actions further include:
inputting into an image analysis model the new image, wherein the image analysis model comprises a plurality of model parameters learned using a set of training data comprising images obtained from a same type of scanner as the second type of scanner;
analyzing, by the image analysis model, the new image;
generating, by the image analysis model, an analysis result based on the analyzing of the new image; and
outputting the analysis result.
19. The computer-program product of claim 18 , wherein the image analysis model is not trained on images obtained from a same type of scanner as the first type of scanner.
20. The computer-program product of claim 16 , wherein:
the generator model and the discriminator model are part of a Generative Adversarial Network (GAN) model:
the GAN model comprises a plurality of model parameters learned using a set of training data comprising one or more pairwise sets of images, wherein each pair of images within the one or more pairwise sets of images comprises a first image generated by the first type of scanner and a second image generated by the second type of scanner; and
wherein the plurality of model parameters are learned using the set of training data based on minimizing a first loss function to train the discriminator model to maximize a probability of the set of training data and a second loss function to train the discriminator model to minimize a probability of a generated image sampled from the generator model and train the generator model to maximize the probability that the discriminator model assigns to the generated image.