IP Library › Granted Patent US 12,332,970
Granted Patent B2
US 12,332,970 · App. 17/867,537 · Granted Jun 17, 2025

Biological image transformation using machine-learning models

Inventors: Herve Marie-Nelly (San Francisco, CA); Jeevaa Velayutham (Puchong, MY)
Assignee: Insitro, Inc.
G06F18/214G06F18/2431G06N3/045G06N3/088G06T7/0012G06T7/10A61B5/7267A61B10/00G06T2207/10056G06T2207/10064G06T2207/10152G06T2207/20081G06T2207/20084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,332,970
App. No.
17/867,537
Granted
Jun 17, 2025
Kind
B2
Abstract

Described are systems and methods for training a machine-learning model to generate image of biological samples, and systems and methods for generating enhanced images of biological samples. The method for training a machine-learning model to generate images of biological samples may include obtaining a plurality of training images comprising a training image of a first type, and a training image of a second type. The method may also include generating, based on the training image of the first type, a plurality of wavelet coefficients using the machine-learning model; generating, based on the plurality of wavelet coefficients, a synthetic image of the second type; comparing the synthetic image of the second type with the training image of the second type; and updating the machine-learning model based on the comparison.

Claims (65)

1. A system for evaluating a treatment with respect to a disease of interest, comprising:

one or more processors;

a memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:

receiving first one or more images depicting one or more untreated biological samples affected by the disease of interest;

receiving second one or more images depicting one or more treated biological samples affected by the disease of interest and treated by the treatment;

inputting the first one or more images into a trained machine-learning model to obtain first one or more transformed images;

inputting the second one or more images into the trained machine-learning model to obtain second one or more transformed images;

comparing the first one or more transformed images and the second one or more transformed images to evaluate the treatment.

2. The system of claim 1 , wherein the one or more programs include instructions for:

receiving third one or more images depicting one or more healthy biological samples not affected by the disease of interest;

inputting the third one or more images into the trained machine-learning model to obtain third one or more transformed images; and

comparing the first one or more transformed images, the second one or more transformed images, and the third one or more transformed images to evaluate the treatment.

3. The system of claim 2 , wherein the first one or more images, the second one or more images, and the third one or more images are bright-field images.

4. The system of claim 2 , wherein the first one or more transformed images, the second one or more transformed images, and the third one or more transformed images are fluorescence images.

5. The system of claim 2 , wherein the first one or more transformed images, the second one or more transformed images, and the third one or more transformed images are phase images.

6. The system of claim 2 , wherein comparing the first one or more transformed images, the second one or more transformed images, and the third one or more transformed images to evaluate the treatment comprises: identifying, in each image, a signal associated with a biomarker.

7. The system of claim 6 , wherein comparing the first one or more transformed images, the second one or more transformed images, and the third one or more transformed images to evaluate the treatment further comprises:

determining a first distribution based on signals of the biomarker in the first one or more transformed images;

determining a second distribution based on signals of the biomarker in the second one or more transformed images; and

determining a third distribution based on signals of the biomarker in the third one or more transformed images.

8. The system of claim 7 , wherein comparing the first one or more transformed images, the second one or more transformed images, and the third one or more transformed images to evaluate the treatment further comprises:

comparing the first distribution, the second distribution, and the third distribution to evaluate the treatment.

9. The system of claim 2 , wherein comparing the first one or more transformed images, the second one or more transformed images, and the third one or more transformed images to evaluate the treatment comprises: determining, for each image, a score indicative of the statement of the disease of interest.

10. The system of claim 9 , wherein comparing the first one or more transformed images, the second one or more transformed images, and the third one or more transformed images to evaluate the treatment further comprises:

determining a first distribution based on scores of the first one or more transformed images;

determining a second distribution based on scores of the second one or more transformed images; and

determining a third distribution based on scores of the third one or more transformed images.

11. The system of claim 10 , wherein comparing the first one or more transformed images, the second one or more transformed images, and the third one or more transformed images to evaluate the treatment further comprises:

comparing the first distribution, the second distribution, and the third distribution to evaluate the treatment.

12. The system of claim 2 , wherein the treatment is a first treatment, the one or more programs include instructions for:

receiving fourth one or more images depicting a fourth one or more treated biological samples affected by the disease of interest and treated by a second treatment;

inputting the fourth one or more images into the trained machine-learning model to obtain fourth one or more transformed images;

comparing the first one or more transformed images, the second one or more transformed images, the third one or more transformed images, and the fourth one or more transformed images to compare the first treatment and the second treatment.

13. The system of claim 12 , wherein the one or more programs include instructions for: selecting a treatment out of the first treatment and the second treatment based on the comparison.

14. The system of claim 13 , wherein the one or more programs include instructions for: administering the selected treatment.

15. The system of claim 13 , wherein the one or more programs include instructions for: providing a medical recommendation based on the selected treatment.

16. The system of claim 1 , wherein the trained machine-learning model is a GAN model comprising a discriminator and a generator.

17. The system of claim 16 , wherein the trained machine-learning model is a conditional GAN model.

18. The system of claim 16 , wherein the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups.

19. The system of claim 18 , wherein each neural network of the plurality of neural networks is configured to generate wavelet coefficients for a respective frequency group.

20. The system of claim 16 , wherein the discriminator is a PatchGAN neural network.

21. The system of claim 1 , wherein the first one or more images and the second one or more images are bright-field images.

22. The system of claim 1 , wherein the first one or more transformed images, and the second one or more transformed images are fluorescence images.

23. The system of claim 1 , wherein the first one or more transformed images and the second one or more transformed images are phase images.

24. The system of claim 1 , wherein comparing the first one or more transformed images and the second one or more transformed images to evaluate the treatment comprises: identifying, in each image, a signal associated with a biomarker.

25. The system of claim 24 , wherein comparing the first one or more transformed images and the second one or more transformed images to evaluate the treatment further comprises:

determining a first distribution based on signals of the biomarker in the first one or more transformed images; and

determining a second distribution based on signals of the biomarker in the second one or more transformed images.

26. The system of claim 25 , wherein comparing the first one or more transformed images and the second one or more transformed images to evaluate the treatment further comprises:

comparing the first distribution and the second distribution to evaluate the treatment.

27. The system of claim 1 , wherein comparing the first one or more transformed images and the second one or more transformed images to evaluate the treatment comprises: determining, for each image, a score indicative of the statement of the disease of interest.

28. The system of claim 1 , wherein the first one or more images comprise a images.

29. A method for evaluating a treatment with respect to a disease of interest, the method comprising:

receiving first one or more images depicting one or more untreated biological samples affected by the disease of interest;

receiving second one or more images depicting one or more treated biological samples affected by the disease of interest and treated by the treatment;

inputting the first one or more images into a trained machine-learning model to obtain first one or more transformed images;

inputting the second one or more images into the trained machine-learning model to obtain second one or more transformed images;

comparing the first one or more transformed images and the second one or more transformed images to evaluate the treatment.

30. A non-transitory computer-readable storage medium storing one or more programs for evaluating a treatment with respect to a disease of interest, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device, cause the electronic device to:

receiving first one or more images depicting one or more untreated biological samples affected by the disease of interest;

receiving second one or more images depicting one or more treated biological samples affected by the disease of interest and treated by the treatment;

inputting the first one or more images into a trained machine-learning model to obtain first one or more transformed images;

inputting the second one or more images into the trained machine-learning model to obtain second one or more transformed images;

comparing the first one or more transformed images and the second one or more transformed images to evaluate the treatment.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 27, 2022
From: MARIE-NELLY, HERVE; VELAYUTHAM, JEEVAA
To: INSITRO, INC.
Reel/Frame 060647/0206 →
Continuity (5)
Division 17480047 · Sep 20, 2021
Continuation PCTUS2021049327 · Sep 7, 2021
Provisional Application 63143707 · Jan 29, 2021
Provisional Application 63075751 · Sep 8, 2020
Related Publication 20220358331A1 · Nov 10, 2022
References Cited (47)
US 11354804B1 · Behrooz · 2022 [cited by examiner]
US 11423256B2 · Marie-Nelly et al. · 2022 [cited by applicant]
US 20160253466A1 · Agaian et al. · 2016 [cited by applicant]
US 20170357844A1 · Comaniciu · 2017 [cited by examiner]
US 20180286040A1 · Sashida · 2018 [cited by examiner]
US 20190384047A1 · Johnson et al. · 2019 [cited by applicant]
US 20210043331A1 · Ozcan et al. · 2021 [cited by applicant]
US 20210304401A1 · Lau et al. · 2021 [cited by applicant]
US 20210325308A1 · Kannan et al. · 2021 [cited by applicant]
US 20220076067A1 · Marie-Nelly et al. · 2022 [cited by applicant]
JP 2009534665A · 2009 [cited by applicant]
JP 2018180635A · 2018 [cited by applicant]
KR 20190095901A · 2019 [cited by applicant]
WO WO2007121454A1 · 2007 [cited by applicant]
WO WO2019180833A1 · 2019 [cited by applicant]
WO WO2019191697A1 · 2019 [cited by applicant]
Bernet et al., (2006). “Quantitative imaging of complex samples in spiral phase contrast microscopy,” Optics Express, 14(9):3792-3805. [cited by applicant]
Christiansen et al., “In silica labeling: Predicting fluorescent labels in unlabeled images”, Cell, Apr. 19, 2018, 173(3): 792-803, pp. 1-37 (Year: 2018). [cited by applicant]
Cooke et al., (2020). “Physics-enhanced machine learning for virtual fluorescence microscopy,” available online at <https://arxiv.org/pdf/2004.04306.pdf>, 12 pages. [cited by applicant]
De Haan et al., “Deep-Learning-Based Image Reconstruction and Enhancement in Optical Microscopy”, Proceedings of the IEEE, vol. 108, Issue 1, Jan. 2020, pp. 30-50 (Year: 2020). [cited by applicant]
Fürhapter et al., (2005). “Spiral phase contrast imaging in microscopy,” Optics Express, 13(3):689-694. [cited by applicant]
Gardner et al., (2018). “CellPaint: Interactive Illustration of Dynamic Mesoscale Cellular Environments,” IEEE Comput Graph Appl., 38(6):51-66, 23 pages. [cited by applicant]
Guo et al., (2019). “Revealing architectural order with quantitative label-free imaging and deep learning,” eLide, 9:e55502, 33 pages. [cited by applicant]
Ham, (2020). “Deep learning accurately stains digital biopsy slides,” available online at <https://medicalxpress.com/news/2020-05-deep-accurately-digital-biopsy.html>, 4 pages. [cited by applicant]
International Search Report and Written Opinion received for International Patent Application No. PCT/US2021/049327 mailed on Mar. 7, 2022, 15 pages. [cited by applicant]
Invitation to Pay Additional Fees and partial Search Report received for International Patent Application No. PCT/US2021/049327 mailed on Jan. 12, 2022, 11 pages. [cited by applicant]
Jesacher et al., (2005). “Shadow effects in spiral phase contrast microscopy,” Phys Rev Lett., 94(23):233902, 4 pages. [cited by applicant]
Jesacher et al., (2007). “Wavefront correction of spatial light modulators using an optical vortex image,” Optics Express, 15(9):5801-5808. [cited by applicant]
Kellman et al., (2015). “Physics-based Learned Design: Optimized Coded-Illumination for Quantitative Phase Imaging,” Journal of Latex class files, 14(8):1-9. [cited by applicant]
Lu et al., (2016). “Quantitative phase imaging and complex field reconstruction by pupil modulation differential phase contrast,” Optics Express, 24(22):25345-25361. [cited by applicant]
Maurer et al., (2008). “Phase contrast microscopy with full numerical aperture illumination,” Optics Express, 16(24):19821-19829. [cited by applicant]
Maurer et al., (2008). “Upgrading a microscope with a spiral phase plate,” J Microsc., 230:134-142. Abstract Only. [cited by applicant]
Maurer et al., (2010). “Depth-of-field-multiplexing in microscopy,” Optics Express, 18(3)3023-3034. [cited by applicant]
McIntyre et al., (2009). “Differential interference contrast imaging using a spatial light modulator,” 34(19):2988-2990. [cited by applicant]
McIntyre et al., (2010). “Quantitative SLM-based differential interference contrast Imaging,” Optics Express, 18(13):14063-14078. [cited by applicant]
Ounkomol et al., (2018). “Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy,” Nature Methods, 15:917-920, 25 pages. [cited by applicant]
Ouyang et al., (2018). “Deep learning massively accelerates super-resolution localization microscopy,” Nature Biotechnology, 36:460-468, 109 pages. [cited by applicant]
Qu et al., (2019). “Wavelet-based Semi-supervised Adversarial Learning for Synthesizing Realistic 7T from 3T MRI,” Med Image Comput Comput Assist Interv., 11767:786-794. [cited by applicant]
Rivenson et al., (2019). “PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning,” Light: Science &Applications, 8:23, 11 pages. [cited by applicant]
Weigert et al., (2018). “Content-aware image restoration: pushing the limits of fluorescence microscopy,” Nature Methods, 15:1090-1097, 17 pages. [cited by applicant]
European Medicines Agency (2001). “ICH Topic E 10 Choice of Control Group in Clinical Trials,” available online at <https://www.ema.europa.eu/en/documents/scientific-guideline/iche-10-choice-control-group-clinical-trial… [cited by applicant]
Extended European Search Report received for Patent Application No. 24221938.4, mailed on Apr. 4, 2025, 8 pages. [cited by applicant]
Lee et al., (2018). “DeepHCS: Bright-Field to Fluorescence Microscopy Image Conversion Using Deep Learning for Label-Free High-Content Screening,” Medical Image Computing And Computer Assisted Intervention—MICCAI, 335-3… [cited by applicant]
Extended European Search Report received for Patent Application No. 24221939.2, mailed on Apr. 15, 2025, 13 pages. [cited by applicant]
Liu et al., (2020). “Deep learning-based holographic polarization microscopy,” arXiv:2007.00741, 20 pages. [cited by applicant]
Rivenson et al., (2019). “Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning,” Nature Biomedical Engineering, 3(6):466-477. [cited by applicant]
Ronneberger et al., (2015). “U-Net: Convolutional Networks for Biomedical Image Segmentation,”arXiv:1505.04597, 8 pages. [cited by applicant]
Cited By (2)
US 12,511,918 US 12,541,983