Image representation learning in digital pathology
Described herein are systems, methods, and programming for analyzing and classifying digital pathology images. Some embodiments include receiving whole slide images (WSIs) and dividing each of the WSIs into tiles. For each WSI, a random subset of the tiles may be selected and augmented views of each of the selected tiles may be generated. For each of the selected tiles, a first convolutional neural network (CNN) may be trained to: generate, using a first one of the augmented views corresponding to the selected tile, a first representation of the selected tile, and predict a second representation of the selected tile to be generated by a second CNN, wherein the second representation is generated based on a second one of the augmented views of the selected tile.
1 . A computer-implemented method for training machine-learning models customized for digital pathology, the method comprising:
receiving a plurality of whole slide images;
dividing each of the whole slide images into a plurality of tiles;
for each of the whole slide images:
selecting a random subset of the tiles;
generating a plurality of augmented views of each of the selected tiles based on the selected random subset of the tiles by:
creating a plurality of random crops of each of the selected tiles, and
identifying a structural feature relevant in digital pathology that is present in the selected random subset of the tiles; and
selecting based on the identified structural feature relevant in digital pathology, from the plurality of random crops, at least two random crops such that the at least two random crops share the feature relevant in digital pathology, and
augmenting the at least two random crops, wherein each of the at least two random crops is augmented based on shifting pixel values of the respective random crop in accordance with a distance vector between a first center of mass of the respective random crop and a second center of mass of a randomly selected tile in a predetermined color space; and
for each of the selected tiles, training a first convolutional neural network (CNN) to learn the structural feature relevant in digital pathology by:
generating, using a first one of the augmented views corresponding to the selected tile, a first representation of the selected tile;
predicting a second representation of the selected tile to be generated by a second CNN, wherein the second representation is generated based on a second one of the augmented views of the selected tile; and
updating one or more weights of the first CNN based on the first representation and the second representation.
2 . The method of claim 1 , wherein the whole slide images are of a plurality of types of tissue stained with hematoxylin and eosin.
3 . The method of claim 1 , wherein augmenting the at least two random crops further comprises at least one of:
randomly cropping the random crop of the at least two random crop, randomly scaling the random crop of the at least two random crop, randomly flipping the random crop of the at least two random crop about a horizontal or vertical axis, randomly rotating and translating the random crop of the at least two random crop, generating random Gaussian blurring of the random crop of the at least two random crop, randomly changing color to grayscale in the random crop of the at least two random crop, or normalizing the random crop of the at least two random crop.
4 . The method of claim 1 , wherein the pixel values of the respective random crop are shifted within a certain range to preserve color information.
5 . The method of claim 1 , wherein a color of the at least two random crops is being translated and augmented to mimic an image with a color of one of the whole slide images, wherein the whole slide images comprise a training set of images.
6 . The method of claim 1 , further comprising:
projecting an image representation of each tile of the random selected subset of tiles to two-dimensional space.
7 . The method of claim 6 , wherein projecting the image representation comprises:
generating an embedding of each tile of the random selected subset of tiles in a feature embedding space, wherein the embedding is represented as an array of numbers each corresponding to a feature of the tile.
8 . The method of claim 1 , wherein at least some of the random subset of tiles selected overlap.
9 . The method of claim 1 , wherein training the first CNN comprises:
minimizing a loss function based on the generated first representation and the predicted second representation.
10 . The method of claim 1 , further comprising:
preparing a biological sample to be scanned by a scanner;
producing a set of sections of the prepared biological sample;
causing one or more sections of the produced sections to be stained via a staining agent;
providing the one or more stained sections to the scanner to obtain a digital image of each stained section; and
classifying, based on the digital image and the first CNN, the biological sample as depicting at least one of a plurality of candidate results, wherein the candidate results represent a plurality of tissue abnormalities.
11 . The method of claim 1 , further comprising:
receiving a liquid biological sample slide;
providing the liquid biological sample slide to a scanner to capture an image of the liquid biological sample slide; and
classifying, based on the digital image and the trained first CNN, the liquid biological sample as depicting at least one of a plurality of candidate results.
12 . The method of claim 1 , further comprising:
obtaining an image of a biological sample from a scanner via a user device, wherein the trained first CNN is configured to classify the image of the biological sample based on additional information provided by a user of the user device.
13 . The method of claim 1 , further comprising:
classifying a digital image depicting a biological sample using the trained first CNN.
14 . The method of claim 1 , further comprising:
obtaining a digital image depicting a biological sample;
dividing the digital image into a set of tiles;
generating, using the first CNN, an embedding of each tile of the set of tiles in a feature embedding space; and
generating, based on each embedding, an output corresponding to a result tile, the result tile being one of the set of tiles.
15 . The method of claim 14 , wherein the embedding is further generated based on contextual information associated with at least one of: (i) each tile or (ii) content depicted by each tile, and the embedding includes at least one of: (i) a feature indicating a size of an object depicted within the digital image or (ii) a feature indicating a density of an objected depicted within the digital image.
16 . The method of claim 14 , further comprising:
receiving a request to identify a whole slide image based on a tile, an identifier of a tile, or an identifier of the whole slide image, wherein the digital image comprises the whole slide image, and the digital image is obtained based on the request.
17 . A non-transitory computer-readable medium storing computer program instructions for training machine-learning models customized for digital pathology, the program instructions that, when executed by one or more processors, effectuate operations comprising:
receiving a plurality of whole slide images;
dividing each of the whole slide images into a plurality of tiles;
for each of the whole slide images:
selecting a random subset of the tiles;
generating a plurality of augmented views of each of the selected tiles based on the selected random subset of the tiles by:
creating a plurality of random crops of each of the selected tiles, and
identifying a structural feature relevant in digital pathology that is present in the selected random subset of the tiles; and
selecting based on the identified structural feature relevant in digital pathology, from the plurality of random crops, at least two random crops such that the at least two random crops share the feature relevant in digital pathology, and
augmenting the at least two random crops, wherein each of the at least two random crops is augmented based on shifting pixel values of the respective random crop in accordance with a distance vector between a first center of mass of the respective random crop and a second center of mass of a randomly selected tile in a predetermined color space; and
for each of the selected tiles, training a first convolutional neural network (CNN) to learn the structural feature relevant in digital pathology by:
generating, using a first one of the augmented views corresponding to the selected tile, a first representation of the selected tile;
predicting a second representation of the selected tile to be generated by a second CNN, wherein the second representation is generated based on a second one of the augmented views of the selected tile; and
updating one or more weights of the first CNN based on the first representation and the second representation.
18 . A system for training machine-learning models customized for digital pathology, comprising:
memory storing computer program instructions;
one or more processors configured to execute the computer program instructions to:
receive a plurality of whole slide images;
divide each of the whole slide images into a plurality of tiles;
for each of the whole slide images:
select a random subset of the tiles;
generate a plurality of augmented views of each of the selected tiles based on the selected random subset of the tiles by:
creating a plurality of random crops of each of the selected tiles, and
identifying a structural feature relevant in digital pathology that is present in the selected random subset of the tiles; and
selecting based on the identified structural feature relevant in digital pathology, from the plurality of random crops, at least two random crops such that the at least two random crops share the feature relevant in digital pathology, and
augmenting the at least two random crops, wherein each of the at least two random crops is augmented based on shifting pixel values of the respective random crop in accordance with a distance vector between a first center of mass of the respective random crop and a second center of mass of a randomly selected tile in a predetermined color space; and
for each of the selected tiles, train a first convolutional neural network (CNN) to learn the structural feature relevant in digital pathology by:
generating, using a first one of the augmented views corresponding to the selected tile, a first representation of the selected tile;
predicting a second representation of the selected tile to be generated by a second CNN, wherein the second representation is generated based on a second one of the augmented views of the selected tile; and
updating one or more weights of the first CNN based on the first representation and the second representation.