Biological image transformation using machine-learning models
Described are systems and methods for training a machine-learning model to generate image of biological samples, and systems and methods for generating enhanced images of biological samples. The method for training a machine-learning model to generate images of biological samples may include obtaining a plurality of training images comprising a training image of a first type, and a training image of a second type. The method may also include generating, based on the training image of the first type, a plurality of wavelet coefficients using the machine-learning model; generating, based on the plurality of wavelet coefficients, a synthetic image of the second type; comparing the synthetic image of the second type with the training image of the second type; and updating the machine-learning model based on the comparison.
1 . A system for generating enhanced images of biological samples, the system comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
providing as input to a machine-learning model at least one label-free microscopic image of a biological sample, wherein the at least one label-free microscopic image requires no exogenous biological or chemical labels or stains of the biological sample, and wherein the at least one label-free microscopic image comprises an image that is not a brightfield image; and
generating, based on the at least one label-free microscopic image of the biological sample, a labeled image of the biological sample using the machine-learning model, wherein the labeled image of the biological sample shows exogenous biological or chemical labels or stains of the biological sample, and wherein the labeled image is selected from the group consisting of: fluorescence images, genetically encoded fluorescent protein images, in situ hybridization images, bioluminescence images, chemiluminescence images, images of fluorophore-conjugated small molecule or peptide probes, nanoparticle-labeled images, semantic map images, absorbance map images, or phospho-specific immunostaining images.
2 . The system of claim 1 , wherein the at least one label-free microscopic image comprises one or more of: phase images, phase-shift images, quantitative phase images, differential interference contrast images, darkfield images, polarized light images, reflection interference contrast images, Raman microscopy images, coherent anti-stokes Raman scattering images, stimulated Raman scattering microscopy images, second-harmonic generation and third-harmonic generation microscopy images, infrared/Fourier transform infrared microscopy images, photoacoustic images, electron microscopy images, atomic force microscopy images, semantic maps, polarization maps, refractive maps (2D and 3D), or absorbance maps.
3 . The system of claim 1 , wherein the fluorescence images comprise one or more of: DAPI images, MAP2 images, pS6 images, or immunofluorescence images.
4 . The system of claim 1 , wherein the fluorescence microscopy images comprise confocal, total internal reflection fluorescence, two-photon, and super-resolution images.
5 . The system of claim 1 , wherein the phospho-specific immunostaining images comprise one or more of: immunofluorescence images with phospho-specific antibodies, western blot images with fluorescent or chemiluminescent detection, images with genetically encoded phosphorylation reporters including FRET or BRET sensors, images with small-molecule fluorescent or bioluminescent probes, proximity ligation images, and immuno-gold labeled electron microscopy images.
6 . The system of claim 1 , wherein the at least one label-free microscopic image comprises images of two or more modalities.
7 . The system of claim 6 , wherein the machine learning model comprises two or more generators.
8 . The system of claim 7 , wherein the two or more generators generate two or more labeled, synthetic images from the label-free images of two or more modalities.
9 . The system of claim 6 , wherein the at least one label-free microscopic image comprises a brightfield image.
10 . The system of claim 1 , wherein the machine-learning model has been trained by:
obtaining a training image of a first type and a training image of a second type, wherein the training image of the first type comprises a phase image;
generating, based on the training image of the first type, a synthetic image of the second type;
comparing the synthetic image of the second type with the training image of the second type; and
updating the machine-learning model based on the comparison.
11 . The system of claim 10 , wherein:
the training image of the first type comprises two or more images of different image modalities;
the machine learning model comprises two or more generators; and
the two or more generators generate two or more synthetic images of the second type from the two or more images of different image modalities of the first type.
12 . The system of claim 10 , wherein the training image of the first type comprises an image captured using a microscope according to a first illumination scheme.
13 . The system of claim 12 , wherein the first illumination scheme comprises one or more illumination patterns.
14 . The system of claim 13 , wherein the one or more programs include instructions for:
based on the comparison, identifying a second illumination scheme;
obtaining at least one additional training image of the first type, wherein the at least one additional training image of the first type comprises at least one image obtained based on the second illumination scheme; and
training the machine-learning model based on the at least one additional training image of the first type.
15 . The system of claim 13 , wherein the one or more programs include instructions for:
obtaining a plurality of images of the first type; and
generating, based on the obtained plurality of images, a plurality of synthetic images of the second type using the machine-learning model.
16 . The system of claim 15 , wherein the one or more programs include instructions for: training a classifier based on the plurality of synthetic images of the second type.
17 . The system of claim 1 , wherein the at least one label-free microscopic image comprises a phase shift image.
18 . The system of claim 17 , wherein each pixel of the phase shift image indicates a local value of a phase in the image.
19 . The system of claim 1 , wherein the machine-learning model comprises a generator and a discriminator.
20 . The system of claim 19 , wherein the machine-learning model comprises a conditional GAN model.
21 . The system of claim 20 , wherein the discriminator is a PatchGAN neural network.
22 . The system of claim 19 , wherein the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups.
23 . The system of claim 22 , wherein each neural network of the plurality of neural networks is configured to generate wavelet coefficients for a respective frequency group.
24 . The system of claim 22 , wherein the plurality of neural networks comprises a plurality of U-Net neural networks.
25 . The system of claim 1 , wherein the in situ hybridization images comprise fluorescence in situ hybridization images.
26 . A method for generating enhanced images of biological samples, the method comprising:
providing as input to a machine-learning model at least one label-free microscopic image of a biological sample, wherein the at least one label-free microscopic image requires no exogenous biological or chemical labels or stains of the biological sample, and wherein the at least one label-free microscopic image comprises an image that is not a brightfield image; and
generating, based on the at least one label-free microscopic image of the biological sample, a labeled image of the biological sample using the machine-learning model, wherein the labeled image of the biological sample shows exogenous biological or chemical labels or stains of the biological sample, and wherein the labeled image is selected from the group consisting of: fluorescence images, genetically encoded fluorescent protein images, in situ hybridization images, bioluminescence images, chemiluminescence images, images of fluorophore-conjugated small molecule or peptide probes, nanoparticle-labeled images, semantic map images, absorbance map images, or phospho-specific immunostaining images.
27 . A non-transitory computer-readable storage medium storing one or more programs for generating enhanced images of biological samples, the one or more programs comprising instructions, which when executed by one or more processors of a system, cause the system to:
provide as input to a machine-learning model at least one label-free microscopic image of a biological sample, wherein the at least one label-free microscopic image requires no exogenous biological or chemical labels or stains of the biological sample, and wherein the at least one label-free microscopic image comprises an image that is not a brightfield image; and
generate, based on the at least one label-free microscopic image of the biological sample, a labeled image of the biological sample using the machine-learning model, wherein the labeled image of the biological sample shows exogenous biological or chemical labels or stains of the biological sample, and wherein the labeled image is selected from the group consisting of: fluorescence images, genetically encoded fluorescent protein images, in situ hybridization images, bioluminescence images, chemiluminescence images, images of fluorophore-conjugated small molecule or peptide probes, nanoparticle-labeled images, semantic map images, absorbance map images, or phospho-specific immunostaining images.