IP Library › Granted Patent US 12,228,722
Granted Patent B2
US 12,228,722 · App. 18/083,182 · Granted Feb 18, 2025

Microscopy system and method for modifying microscope images in the feature space of a generative network

Inventors: Manuel Amthor (Jena, DE); Daniel Haase (Zoellnitz, DE)
Assignee: Carl Zeiss Microscopy GmbH
G02B21/367G06T7/0012G06T2207/10056G06T2207/20076G06T2207/20081
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,228,722
App. No.
18/083,182
Granted
Feb 18, 2025
Kind
B2
Abstract

In a computer-implemented method for modifying microscope images, a generative model is trained using a training dataset which comprises a plurality of microscope images. After the training, the generative model is configured to compute a generated microscope image from a feature vector derived from a feature space. It is established which image properties are affected by which feature variables in the feature space. A microscope image to be modified is received and projected into the feature space in order to obtain an associated feature vector. One or more feature variables of the feature vector are modified in order to change one or more image properties, whereby a modified feature vector is generated. The modified feature vector is projected back into an image space by inputting the modified feature vector into the generative model, thereby generating a modified microscope image.

Claims (67)

1. A computer-implemented method for modifying microscope images, comprising:

training a generative model using a training dataset which comprises a plurality of microscope images, wherein after the training the generative model is configured to calculate a generated microscope image from a feature vector derived from a feature space;

establishing which image properties are affected by which feature variables in the feature space;

receiving a microscope image to be modified;

projecting the microscope image to be modified into the feature space in order to obtain an associated feature vector;

generating a modified feature vector by modifying one or more feature variables of the feature vector in order to change one or more image properties; and

generating a modified microscope image from projecting the modified feature vector back into an image space by inputting the modified feature vector into the generative model.

2. The computer-implemented method according to claim 1 ,

wherein the generative model is formed by a generator of a generative adversarial network,

wherein the modified microscope image is input into a discriminator of the generative adversarial network, wherein whether image artefacts were caused by the modification of the feature vector is inferred as a function of an output of the discriminator.

3. The computer-implemented method according to claim 1 ,

wherein the generative model is formed by a generator of generative adversarial networks or by a decoder of an autoencoder.

4. The computer-implemented method according to claim 1 ,

wherein the one or more image properties relate to one or more of the following:

an exposure of an image or image area;

a contamination of a sample carrier or cover slip area;

a contrast of cover slip edges;

reflections, local dimming or other artefacts on a sample carrier;

background artefacts visible through a transparent sample carrier; and

a background illumination.

5. The computer-implemented method according to claim 1 , further comprising:

providing a selection option with which a user can specify an intended change in the at least one image property, wherein one or more feature variables of the feature vector are modified in accordance with the intended change.

6. The computer-implemented method according to claim 1 ,

wherein one or more of the feature variables of the feature vector are automatically modified in accordance with predetermined criteria.

7. The computer-implemented method according to claim 1 , further comprising:

inputting the modified microscope image into a trained inspection model or into an inspection program configured to establish whether image artefacts were caused by the modification of the feature vector.

8. A computer-implemented method of calculating an image processing result for an input image,

wherein an image processing program calculates the image processing result for the input image by first:

carrying out the method according to claim 1 , wherein the modification of one or more feature variables of the feature vector occurs in accordance with requirements of the image processing program,

and subsequently:

inputting the modified microscope image into the image processing program, which calculates the image processing result therefrom.

9. A computer-implemented method for modifying microscope images, comprising:

receiving a microscope image to be modified;

generating a modified microscope image using a generative model, wherein the generative model has been trained, using a training dataset which comprises a plurality of microscope images, to calculate a generated microscope image from a feature vector derived from a feature space, wherein at least the following steps are performed to generate the modified microscope image using the generative model:

projecting the microscope image to be modified into the feature space in order to obtain an associated feature vector;

generating a modified feature vector by modifying one or more feature variables of the feature vector in order to change one or more image properties; and

generating the modified microscope image from projecting the modified feature vector back into an image space by inputting the modified feature vector into the generative model.

10. The computer-implemented method according to claim 9 ,

wherein the generative model is formed by a generator of generative adversarial networks or by a decoder of an autoencoder.

11. The computer-implemented method according to claim 9 ,

wherein the one or more image properties relate to one or more of the following:

an exposure of an image or image area;

a contamination of a sample carrier or cover slip area;

a contrast of cover slip edges;

reflections, local dimming or other artefacts on a sample carrier;

background artefacts visible through a transparent sample carrier; and

a background illumination.

12. The computer-implemented method according to claim 9 , further comprising:

providing a selection option with which a user can specify an intended change in the at least one image property, wherein one or more feature variables of the feature vector are modified in accordance with the intended change.

13. The computer-implemented method according to claim 9 ,

wherein one or more of the feature variables of the feature vector are automatically modified in accordance with predetermined criteria.

14. The computer-implemented method according to claim 9 , further comprising:

inputting the modified microscope image into a trained inspection model or into an inspection program configured to establish whether image artefacts were caused by the modification of the feature vector.

15. The computer-implemented method according to claim 9 ,

wherein the generative model is formed by a generator of a generative adversarial network,

wherein the modified microscope image is input into a discriminator of the generative adversarial network, wherein it whether image artefacts were caused by the modification of the feature vector is inferred as a function of an output of the discriminator.

16. A computer-implemented method of calculating an image processing result for an input image,

wherein an image processing program calculates the image processing result for the input image by first:

carrying out the method according to claim 9 , wherein the modification of one or more feature variables of the feature vector occurs in accordance with requirements of the image processing program,

and subsequently:

inputting the modified microscope image into the image processing program, which calculates the image processing result therefrom.

17. A microscopy system, comprising:

a microscope for capturing microscope images; and

a computing device which is configured to execute the method according to claim 1 .

18. A computer-readable storage medium, comprising commands which, when executed by a computer, cause the computer to execute the method according to claim 1 .

19. The computer-implemented method according to claim 8 , wherein the image processing program is a segmentation model, a detection model, a classification model, or a model for image-to-image mapping.

20. The computer-implemented method according to claim 16 , wherein the image processing program is a segmentation model, a detection model, a classification model, or a model for image-to-image mapping.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 19, 2023
From: AMTHOR, MANUEL; HAASE, DANIEL, DR.
To: CARL ZEISS MICROSCOPY GMBH
Reel/Frame 063383/0421 →
Priority Claims (1)
DE 10 2021 133 868.9 · Dec 20, 2021 · national
Continuity (1)
Related Publication 20230194847A1 · Jun 22, 2023
References Cited (18)
US 20050013478A1 · Oba · 2005 [cited by examiner]
US 20220130126A1 · Delgado · 2022 [cited by examiner]
US 20220375047A1 · Gurevich · 2022 [cited by examiner]
Xia et al., “GAN Inversion: A Survey,” available at https://arxiv.org/abs/2101.05278, Mar. 22, 2022, 17 pages. [cited by applicant]
Voynov et al., “Unsupervised Discovery of Interpretable Directions in the GAN Latent Space,” available at https://arxiv.org/abs/2002.03754, Jun. 24, 2020, 15 pages. [cited by applicant]
Ling et al., “EditGAN: High-Precision Semantic Image Editing,” available at https://arxiv.org/abs/2111.03186, Nov. 4, 2021, 38 pages. [cited by applicant]
Gu et al., “GIQA: Generated Image Quality Assessment,” available at https://arxiv.org/abs/2003.08932, Jul. 14, 2020, 26 pages. [cited by applicant]
Mikhailiuk, Aliaksei, “Deep Image Quality Assessment—Towards Data Science,” https://towardsdatascience.com/deep-image-quality-assessment-30ad71641fac, Mar. 15, 2021, 19 pages. [cited by applicant]
Sertis, “GAN Inversion: A brief walkthrough—Part I—Sertis—Medium,” https://sertiscorp.medium.com/gan-inversion-a-brief-walkthrough-part-i-bc2ee1b73253, Nov. 8, 2021, 12 pages. [cited by applicant]
Goodfellow et al., “Generative Adversarial Nets,” available at https://arxiv.org/abs/1406.2661, Jun. 10, 2014, 9 pages. [cited by applicant]
Gao et al., “A Review of Active Appearance Models,” IEE Transactions on Systems, Man, and Cybernetics—Part C: Applications and Reviews, vol. 40, No. 2, Mar. 2010, pp. 145-158. [cited by applicant]
Kaust et al., “Image2StyleGAN++: How to Edit the Embedded Images?” available at https://arxiv.org/abs/1911.11544, Aug. 7, 2020, 18 pages. [cited by applicant]
Shen et al., “Interpreting the Latent Space of GANs for Semantic Face Editting,” available at https://arxiv.org/abs/1907.10786, Mar. 31, 2020, 12 pages. [cited by applicant]
Nguyen et al., “Deep Learning for Deepfakes Creation and Detection: A Survey,” available at https://arxiv.org/abs/1909.11573v5, Aug. 11, 2022, 19 pages. [cited by applicant]
Karras et al., “A Style-Based Generator Architecture for Generative Adversarial Networks” available at https://arxiv.org/abs/1812.04948, Mar. 29, 2019, 12 pages. [cited by applicant]
Karras, T., et al., “Analyzing and Improving the Image Quality of StyleGAN” available at https://arxiv.org/abs/1912.04958, Mar. 23, 2020, 21 pages. [cited by applicant]
Kaust et al., “Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?” available at https://arxiv.org/abs/1904.03189, Sep. 3, 2019, 23 pages. [cited by applicant]
Cootes et al., “Active Appearance Models,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, No. 6, Jun. 2001, pp. 681-685. [cited by applicant]