Biological image transformation using machine-learning models
Described are systems and methods for training a machine-learning model to generate image of biological samples, and systems and methods for generating enhanced images of biological samples. The method for training a machine-learning model to generate images of biological samples may include obtaining a plurality of training images comprising a training image of a first type, and a training image of a second type. The method may also include generating, based on the training image of the first type, a plurality of wavelet coefficients using the machine-learning model; generating, based on the plurality of wavelet coefficients, a synthetic image of the second type; comparing the synthetic image of the second type with the training image of the second type; and updating the machine-learning model based on the comparison.
1 . A system for generating enhanced images of biological samples, the system comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
providing as input to a trained machine-learning model at least two microscopic images of a biological sample, wherein the at least two microscopic images of the biological sample comprise a first image comprising a phase image and a second image comprising a bright-field image, a fluorescence image, a semantic map, a polarization map, a refractive map, an absorbance map, or any combination thereof; and
generating, based on the at least two microscopic images of the biological sample, an enhanced image of the biological sample using the trained machine-learning model.
2 . The system of claim 1 , wherein the trained machine-learning model has been trained by:
obtaining a training image of a first type and a training image of a second type, wherein the training image of the first type comprises a training phase image;
generating, based on the training image of the first type, a synthetic image of the second type;
comparing the synthetic image of the second type with the training image of the second type; and
updating the trained machine-learning model based on the comparison.
3 . The system of claim 2 , wherein the training image of the first type comprises an image captured using a microscope according to a first illumination scheme.
4 . The system of claim 3 , wherein the first illumination scheme comprises one or more illumination patterns.
5 . The system of claim 3 , wherein the one or more programs include instructions for:
based on the comparison, identifying a second illumination scheme;
obtaining at least one additional training image of the first type, wherein the at least one additional training image of the first type comprises at least one image obtained based on the second illumination scheme; and
training the trained machine-learning model based on the at least one additional training image of the first type.
6 . The system of claim 3 , wherein the one or more programs include instructions for:
obtaining a plurality of images of the first type; and
generating, based on the obtained plurality of images, a plurality of synthetic images of the second type using the trained machine-learning model.
7 . The system of claim 6 , wherein the one or more programs include instructions for: training a classifier based on the plurality of synthetic images of the second type.
8 . The system of claim 1 , wherein the enhanced image of the biological sample comprises at least one of a bright-field image, a fluorescence image, a semantic map, a polarization map, a refractive map, and an absorbance map.
9 . The system of claim 8 , wherein the enhanced image of the biological sample comprises a fluorescence image.
10 . The system of claim 1 , wherein the phase image comprises a phase shift image.
11 . The system of claim 10 , wherein each pixel of the phase shift image indicates a local value of a phase in the image.
12 . The system of claim 1 , wherein the trained machine-learning model comprises a generator and a discriminator.
13 . The system of claim 12 , wherein the trained machine-learning model comprises a conditional GAN model.
14 . The system of claim 13 , wherein the discriminator is a PatchGAN neural network.
15 . The system of claim 12 , wherein the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups.
16 . The system of claim 15 , wherein each neural network of the plurality of neural networks is configured to generate wavelet coefficients for a respective frequency group.
17 . The system of claim 15 , wherein the plurality of neural networks comprises a plurality of U-Net neural networks.
18 . The system of claim 1 , wherein the instructions for providing input to the trained machine-learning model comprise instructions for providing the phase image and the bright-field image to the trained machine-learning model.
19 . The system of claim 1 , wherein the instructions for providing input to the trained machine-learning model comprise instructions for providing to the trained machine-learning model (a) the phase image and (b) the bright-field image, the fluorescence image, the semantic map, the refractive map, or the absorbance map.
20 . A method for generating enhanced images of biological samples, the method comprising:
providing as input to a trained machine-learning model at least two microscopic images of a biological sample, wherein the at least two microscopic images of the biological sample comprise a first image comprising a phase image and a second image comprising a bright-field image, a fluorescence image, a semantic map, a polarization map, a refractive map, an absorbance map, or any combination thereof; and
generating, based on the at least two microscopic images of the biological sample, an enhanced image of the biological sample using the trained machine-learning model.
21 . A non-transitory computer-readable storage medium storing one or more programs for generating enhanced images of biological samples, the one or more programs comprising instructions, which when executed by one or more processors of a system, cause the system to:
provide as input to a trained machine-learning model at least two microscopic images of a biological sample, wherein the at least two microscopic images of the biological sample comprise a first image comprising a phase image and a second image comprising a bright-field image, a fluorescence image, a semantic map, a polarization map, a refractive map, an absorbance map, or any combination thereof; and
generate, based on the at least two microscopic images of the biological sample, an enhanced image of the biological sample using the trained machine-learning model.