IP Library › Granted Patent US 12,511,918
Granted Patent B2
US 12,511,918 · App. 19/006,979 · Granted Dec 30, 2025

Biological image transformation using machine-learning models

Inventors: Herve Marie-Nelly (San Francisco, CA); Jeevaa Velayutham (Shah Alam, MY)
Assignee: Insitro, Inc.
G06V20/69G06F18/214G06F18/2431G06N3/045G06N3/088G06T5/60G06T7/0012G06T7/10A61B5/7267A61B10/00G06T2207/10056G06T2207/10064G06T2207/10152G06T2207/20081G06T2207/20084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,511,918
App. No.
19/006,979
Granted
Dec 30, 2025
Kind
B2
Abstract

Described are systems and methods for training a machine-learning model to generate image of biological samples, and systems and methods for generating enhanced images of biological samples. The method for training a machine-learning model to generate images of biological samples may include obtaining a plurality of training images comprising a training image of a first type, and a training image of a second type. The method may also include generating, based on the training image of the first type, a plurality of wavelet coefficients using the machine-learning model; generating, based on the plurality of wavelet coefficients, a synthetic image of the second type; comparing the synthetic image of the second type with the training image of the second type; and updating the machine-learning model based on the comparison.

Claims (59)

1 . A system for generating enhanced images of biological samples, the system comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:

generating, using a microscopic image of a biological sample of a first type, an enhanced image of the biological sample of a second type using a machine-learning model, wherein the machine-learning model has been trained by:

receiving a training image of the first type and a training image of the second type, wherein the training image of the first type is part of a bright-field image array of a biological sample, wherein the training image of the first type is captured using a microscope according to a first illumination scheme;

generating, based on the training image of the first type, a synthetic image of the second type;

comparing the synthetic image of the second type with the training image of the second type; and

updating the machine-learning model based on the comparison.

2 . The system of claim 1 , wherein the enhanced image is a phase shift image.

3 . The system of claim 2 , wherein one or more pixels of the phase shift image indicate a local value of a phase in the image.

4 . The system of claim 1 , wherein the machine-learning model comprises a generator and a discriminator.

5 . The system of claim 4 , wherein the machine-learning model comprises a conditional GAN model.

6 . The system of claim 5 , wherein the discriminator is a PatchGAN neural network.

7 . The system of claim 4 , wherein the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups.

8 . The system of claim 7 , each neural network of the plurality of neural networks is configured to generate wavelet coefficients for a respective frequency group.

9 . The system of claim 7 , wherein the plurality of neural networks comprises a plurality of U-Net neural networks.

10 . The system of claim 1 , wherein the first illumination scheme comprises one or more illumination patterns.

11 . The system of claim 1 , wherein the one or more programs include instructions for:

based on the comparison, identifying a second illumination scheme;

obtaining at least one additional training image of the first type, wherein the at least one additional training image of the first type is obtained based on the second illumination scheme;

training the machine-learning model based on the at least one additional training image of the first type.

12 . The system of claim 1 , wherein the one or more programs include instructions for:

obtaining, using a microscope, a plurality of images of the first type; and

generating, based on the obtained plurality of images, a plurality of synthetic images of the second type using the machine-learning model.

13 . The system of claim 12 , wherein the one or more programs include instructions for: training a classifier based on the plurality of synthetic images of the second type.

14 . The system of claim 12 , wherein the microscope is a first microscope, wherein the classifier is a first classifier, further comprising:

obtaining, using a second microscope, a plurality of images of the second type;

training a second classifier based on the plurality of images of the second type; and

comparing performance of the first classifier and the second classifier.

15 . The system of claim 14 , wherein the plurality of images of the second type comprise phase images.

16 . The system of claim 1 , wherein the training image of the second type is a phase image.

17 . A method for generating enhanced images of biological samples, the method comprising:

generating, using a microscopic image of a biological sample of a first type, an enhanced image of the biological sample of a second type using a machine-learning model, wherein the machine-learning model has been trained by:

receiving a training image of the first type and a training image of the second type, wherein the training image of the first type is part of a bright-field image array of a biological sample, wherein the training image of the first type is captured using a microscope according to a first illumination scheme;

generating, based on the training image of the first type, a synthetic image of the second type;

comparing the synthetic image of the second type with the training image of the second type; and

updating the machine-learning model based on the comparison.

18 . A non-transitory computer-readable storage medium storing one or more programs for generating enhanced images of biological samples, the one or more programs comprising instructions, which when executed by one or more processors of a system, cause the system to:

generate, using a microscopic image of a biological sample of a first type, an enhanced image of the biological sample of a second type using a machine-learning model, wherein the machine-learning model has been trained by:

receive a training image of the first type and a training image of the second type, wherein the training image of the first type is part of a bright-field image array of a biological sample, wherein the training image of the first type is captured using a microscope according to a first illumination scheme;

generate, based on the training image of the first type, a synthetic image of the second type;

compare the synthetic image of the second type with the training image of the second type; and

update the machine-learning model based on the comparison.

19 . A system for generating enhanced images of biological samples, the system comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:

generating, using a microscopic image of a biological sample of a first type, an enhanced image of the biological sample of a second type using a machine-learning model, wherein the machine-learning model has been trained by:

receiving a training image of the first type and a training image of the second type, wherein the training image of the first type is captured using a microscope according to a first illumination scheme;

generating, based on the training image of the first type, a synthetic image of the second type;

comparing the synthetic image of the second type with the training image of the second type; and

updating the machine-learning model based on the comparison,

based on the comparison, identifying a second illumination scheme;

obtaining at least one additional training image of the first type, wherein the at least one additional training image of the first type is obtained based on the second illumination scheme;

training the machine-learning model based on the at least one additional training image of the first type.

20 . A system for generating enhanced images of biological samples, the system comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:

generating, using a microscopic image of a biological sample of a first type, an enhanced image of the biological sample of a second type using a machine-learning model, and wherein the machine-learning model has been trained by:

obtaining a first plurality of images of the first type;

generating, based on the first plurality of images, a first plurality of synthetic images of the second type using the machine-learning model;

training a first classifier based on the first plurality of synthetic images of the second type;

obtaining a second plurality of images of the second type;

training a second classifier based on the second plurality of images of the second type;

comparing performance of the first classifier and the second classifier; and

updating the machine-learning model based on the comparison.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 15, 2025
From: MARIE-NELLY, HERVE; VELAYUTHAM, JEEVAA
To: INSITRO, INC.
Reel/Frame 069885/0799 →
Continuity (6)
Continuation 17867537 · Jul 18, 2022
Division 17480047 · Sep 20, 2021
Continuation PCTUS2021049327 · Sep 7, 2021
Provisional Application 63143707 · Jan 29, 2021
Provisional Application 63075751 · Sep 8, 2020
Related Publication 20250217443A1 · Jul 3, 2025
References Cited (80)
US 7480640B1 · Elad et al. · 2009 [cited by applicant]
US 11354804B1 · Behrooz et al. · 2022 [cited by applicant]
US 11423256B2 · Marie-Nelly et al. · 2022 [cited by applicant]
US 12332970B2 · Marie-Nelly et al. · 2025 [cited by applicant]
US 20140233826A1 · Agaian et al. · 2014 [cited by applicant]
US 20160253466A1 · Agaian et al. · 2016 [cited by applicant]
US 20170357844A1 · Comaniciu et al. · 2017 [cited by applicant]
US 20180286040A1 · Sashida · 2018 [cited by applicant]
US 20190051057A1 · Sommerlade et al. · 2019 [cited by applicant]
US 20190294108A1 · Ozcan · 2019 [cited by examiner]
US 20190333199A1 · Ozcan et al. · 2019 [cited by applicant]
US 20190384047A1 · Johnson et al. · 2019 [cited by applicant]
US 20200167546A1 · Herring · 2020 [cited by examiner]
US 20210043331A1 · Ozcan et al. · 2021 [cited by applicant]
US 20210097661A1 · Palo · 2021 [cited by examiner]
US 20210158521A1 · Shaked et al. · 2021 [cited by applicant]
US 20210190762A1 · Boehm et al. · 2021 [cited by applicant]
US 20210264214A1 · Ozcan et al. · 2021 [cited by applicant]
US 20210304401A1 · Lau et al. · 2021 [cited by applicant]
US 20210325308A1 · Kannan et al. · 2021 [cited by applicant]
US 20220012850A1 · Ozcan · 2022 [cited by examiner]
US 20220043251A1 · Moore · 2022 [cited by examiner]
US 20220076067A1 · Marie-Nelly et al. · 2022 [cited by applicant]
US 20220206434A1 · Ozcan · 2022 [cited by examiner]
US 20220358331A1 · Marie-Nelly et al. · 2022 [cited by applicant]
US 20230030424A1 · Ozcan · 2023 [cited by examiner]
US 20250139200A1 · Marie-Nelly et al. · 2025 [cited by applicant]
US 20250139201A1 · Marie-Nelly et al. · 2025 [cited by applicant]
JP 2009534665 · 2009 [cited by applicant]
JP 2009534665A · 2009 [cited by applicant]
JP 2018180635A · 2018 [cited by applicant]
KR 101084719B1 · 2011 [cited by applicant]
KR 20120028760A · 2012 [cited by applicant]
KR 20190095901A · 2019 [cited by applicant]
WO WO2007121454A1 · 2007 [cited by applicant]
WO WO2019028269A2 · 2019 [cited by applicant]
WO WO2019180833A1 · 2019 [cited by applicant]
WO WO2019191697A1 · 2019 [cited by applicant]
Nygate et al., “Holographic virtual staining of individual biological cells”, Proc. Natl. Acad. Sci. U.S.A. 117 (17), Apr. 28, 2020, pp. 9223-9231. (Year: 2020). [cited by examiner]
Wu et al., “Bright-field holography: cross-modality deep learning enables snapshot 3D imaging with bright-field contrast using a single hologram”, Ligh: Science and Applications (2019)8:25, pp. 1-7. (Year: 2019). [cited by examiner]
Go et al., “Deep learning-based hologram generation using a white light source”, Sci Rep 10, 8977 (2020), pp. 1-12. (Year: 2020). [cited by examiner]
European Medicines Agency (2001). “ICH Topic E 10 Choice of Control Group in Clinical Trials,” available online at <https://www.ema.europa.eu/en/documents/scientific-guideline/iche-10-choice-control-group-clinical-trial… [cited by applicant]
Extended European Search Report received for Patent Application No. 24221938.4, mailed on Apr. 4, 2025, 8 pages. [cited by applicant]
Lee et al., (2018). “DeepHCS: Bright-Field to Fluorescence Microscopy Image Conversion Using Deep Learning for Label-Free High-Content Screening,” Medical Image Computing And Computer Assisted Intervention—MICCAI, 335-3… [cited by applicant]
Bernet et al., (2006). “Quantitative imaging of complex samples in spiral phase contrast microscopy,” Optics Express, 14(9):3792-3805. [cited by applicant]
Christiansen et al., “In silica labeling: Predicting fluorescent labels in unlabeled images”, Cell, Apr. 19, 2018, 173(3): 792-803, pp. 1-37 (Year: 2018). [cited by applicant]
Cooke et al., (2020). “Physics-enhanced machine learning for virtual fluorescence microscopy,” available online at <https://arxiv.org/pdf/2004.04306.pdf>, 12 pages. [cited by applicant]
De Haan et al., “Deep-Learning-Based Image Reconstruction and Enhancement in Optical Microscopy”, Proceedings of the IEEE, vol. 108, Issue 1, Jan. 2020, pp. 30-50 (Year: 2020). [cited by applicant]
Fürhapter et al., (2005). “Spiral phase contrast imaging in microscopy,” Optics Express, 13(3):689-694. [cited by applicant]
Gardner et al., (2018). “CellPAINT: Interactive Illustration of Dynamic Mesoscale Cellular Environments,” IEEE Comput Graph Appl., 38(6):51-66, 23 pages. [cited by applicant]
Guo et al., (2019). “Revealing architectural order with quantitative label-free imaging and deep learning,” eLide, 9:e55502, 33 pages. [cited by applicant]
Ham, (2020). “Deep learning accurately stains digital biopsy slides,” available online at <https://medicalxpress.com/news/2020-05-deep-accurately-digital-biopsy.html>, 4 pages. [cited by applicant]
International Search Report and Written Opinion received for International Patent Application No. PCT/US2021/049327 mailed on Mar. 7, 2022, 14 pages. [cited by applicant]
Invitation to Pay Additional Fees and partial Search Report received for International Patent Application No. PCT/US2021/049327 mailed on Jan. 12, 2022, 11 pages. [cited by applicant]
Jesacher et al., (2005). “Shadow effects in spiral phase contrast microscopy,” Phys Rev Lett., 94(23):233902, 4 pages. [cited by applicant]
Jesacher et al., (2007). “Wavefront correction of spatial light modulators using an optical vortex image,” Optics Express, 15(9):5801-5808. [cited by applicant]
Kellman et al., (2015). “Physics-based Learned Design: Optimized Coded-Illumination for Quantitative Phase Imaging,” Journal of Latex class files, 14(8):1-9. [cited by applicant]
Lu et al., (2016). “Quantitative phase imaging and complex field reconstruction by pupil modulation differential phase contrast,” Optics Express, 24(22):25345-25361. [cited by applicant]
Maurer et al., (2008). “Phase contrast microscopy with full numerical aperture illumination,” Optics Express, 16(24):19821-19829. [cited by applicant]
Maurer et al., (2008). “Upgrading a microscope with a spiral phase plate,” J Microsc., 230:134-142. Abstract Only. [cited by applicant]
Maurer et al., (2010). “Depth-of-field-multiplexing in microscopy,” Optics Express, 18(3)3023-3034. [cited by applicant]
McIntyre et al., (2009). “Differential interference contrast imaging using a spatial light modulator,” Opt Lett. 34(19):2988-2990. [cited by applicant]
McIntyre et al., (2010). “Quantitative SLM-based differential interference contrast Imaging,” Optics Express, 18(13):14063-14078. [cited by applicant]
Ounkomol et al., (2018). “Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy,” Nature Methods, 15:917-920, 25 pages. [cited by applicant]
Ouyang et al., (2018). “Deep learning massively accelerates super-resolution localization microscopy,” Nature Biotechnology, 36:460-468, 109 pages. [cited by applicant]
Qu et al., (2019). “Wavelet-based Semi-supervised Adversarial Learning for Synthesizing Realistic 7T from 3T MRI,” Med Image Comput Comput Assist Interv., 11767:786-794. [cited by applicant]
Rivenson et al., (2019). “PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning,” Light: Science &Applications, 8:23, 11 pages. [cited by applicant]
Weigert et al., (2018). “Content-aware image restoration: pushing the limits of fluorescence microscopy,” Nature Methods, 15:1090-1097, 17 pages. [cited by applicant]
Extended European Search Report received for Patent Application No. 24221939.2, mailed on Apr. 15, 2025, 13 pages. [cited by applicant]
Liu et al., (2020). “Deep learning-based holographic polarization microscopy,” arXiv:2007.00741, 20 pages. [cited by applicant]
Rivenson et al., (2019). “Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning,” Nature Biomedical Engineering, 3(6):466-477, 12 pages. [cited by applicant]
Ronneberger et al., (2015). “U-Net: Convolutional Networks for Biomedical Image Segmentation, ”arXiv:1505.04597, 8 pages. [cited by applicant]
Van Valen et al., (2016). “Deep Learning Automates the Quantitative Analysis of Individual Cells in Live-Cell Imaging Experiments,” PLOS Computational Biology, 12(11):e1005177, 24 pages. [cited by applicant]
KR Office Action dated May 28, 2025 in KR Application No. 1020237011949 with English Translation. [cited by applicant]
IL Office Action dated May 26, 2025 in IL Application No. 301182. [cited by applicant]
IN Office Action dated Aug. 8, 2025 in IN Application No. 202317025749. [cited by applicant]
Qu liangqiong et al, “Wavelet based semi supervised adversial learning for synthesis realistic 7T from 3T MRI”, Publication date—Oct. 10, 2019. [cited by applicant]
Final Office Action, U.S. Appl. No. 19/007,119, dated Jul. 29, 2025, 31 pages. [cited by applicant]
Notice of Allowance, U.S. Appl. No. 19/007,124, dated Jun. 24, 2025, 8 pages. [cited by applicant]
Kim Y., et al., “High-Performance of Deep Learning Colorization With Wavelet Fusion,” IEMEK Journal of Embedded Systems and Applications, 2018, vol. 13 (6), pp. 313-319. [cited by applicant]