Systems and methods for image preprocessing
A method and apparatus of a device that classifies an image is described. In an exemplary embodiment, the device segments the image into a region of interest that includes information useful for classification and a background region by applying a first convolutional neural network. In addition, the device tiles the region of interest into a set of tiles. For each tile, the device extracts a feature vector of that tile by applying a second convolutional neural network, where the features of the feature vectors represent local descriptors of the tile. Furthermore, the device processes the extracted feature vectors of the set of tiles to classify the image.
1. A method of classifying an input image, the method comprising:
segmenting the input image into a region of interest that includes information useful for classification and a background region by applying a first convolutional neural network;
tiling the region of interest into a set of tiles;
for each tile, extracting a feature vector of that tile by applying a second convolutional neural network, wherein the features of the feature vectors represent local descriptors of the tile and a score is associated with each tile in the set of tiles; and
processing the extracted feature vectors of the set of tiles to classify the input image, including using a first subset of the set of tiles having highest scores and a second subset of the set of tiles having lowest scores and not using any remaining tiles of the set of tiles, as input to a classifier neural network.
2. The method according to claim 1 , wherein said first convolutional network is a semantic segmentation neural network classifying each pixel of the input image as one of region of interest and a background region.
3. The method according to claim 1 , wherein the tiling comprises:
applying a fixed tiling grid to at least the region of interest, wherein each of the set of tiles has a predetermined size.
4. The method according to claim 1 , wherein the tiling further comprises:
augmenting the set of tiles by performing an action on the set of tiles, wherein the action is at least one of a rotation, translation, cropping, adding noise to the input image, modifying an intensity of one or more colors, or changing a contrast of the input image.
5. The method according to claim 1 , wherein the tiling further comprises:
normalizing a number of tiles in the set of tiles per the input image by performing at least one of a random sampling of the set of tiles and padding the set of tiles with blank tiles, so that said set of tiles comprises a given number of tiles per the input image.
6. The method according to claim 1 , further comprising:
pre-training weights of the second convolutional neural network on at least one of another dataset and a task, wherein at least one output layer is removed from the second convolutional neural network.
7. The method according to claim 1 , wherein the processing comprises:
for each tile,
computing the score associated with that tile from the extracted feature vector, wherein said tile score represents a contribution of that tile to the classification of the input image;
sorting a set of the tile scores;
selecting the first subset having the highest scores and the second subset having the lowest scores based on at least one of a value and a rank of a tile in the sorted set of tile scores, and
applying the classifier neural network to the subset of tile scores to classify the image.
8. The method according to claim 1 , wherein the tiling further comprises:
aggregating groups of corresponding tiles from the different ones of a plurality images, wherein the plurality of images includes the input image.
9. The method according to claim 1 , wherein the tiling further comprises:
aggregating clusters of tiles from the set of tiles that are within a given distance to each other according to a distance metric computed in at least one of the input image or a feature map.
10. The method according to claim 9 , wherein aggregating a cluster of tiles comprises at least one of:
concatenating tiles in the cluster of tiles;
selecting a single tile from the cluster of tiles based on at least a given criterion, using the cluster of tiles as a multidimensional object for the feature vector extraction;
computing a mean of the feature vectors of tiles in the cluster of tiles; and
computing at least the maximum or minimum value of the feature vectors of tiles in the cluster of tiles.
11. The method according to claim 1 , wherein the feature vector extraction further comprises:
applying an autoencoder on the extracted feature vectors to reduce the dimensionality of the features.
12. The method according to claim 1 , wherein the input image is a histopathology slide and said region of interest is a tissue region.
13. The method according to claim 1 , wherein the image classification is used to perform at least one of a diagnosis classification, survival prediction, and response to treatment prediction.
14. The method according to claim 1 , wherein the classification of the input image includes predicting at least one type of global and local label.
15. The method according to claim 1 , wherein the processing of the extracted feature vectors includes using one or more local annotations associated with the input image that are incorporated as weights in a weighted combination of a feature vector derived from extracted feature vectors that provides additional information for the classification of the input image.
16. The method according to claim 1 , wherein the classification of the input image includes predicting multiple global labels in a multi-task environment.
17. A non-transitory machine readable medium having executable instructions to cause one or more processing units to perform a method to classify an input image, the method comprising:
segmenting the input image into a region of interest that includes information useful for classification and a background region by applying a first convolutional neural network;
tiling said region of interest into a set of tiles;
for each tile, extracting a feature vector of that tile by applying a second convolutional neural network, wherein the features of the feature vectors represent local descriptors of the tile and a score is associated with each tile in the set of tiles; and
processing the extracted feature vectors of the set of tiles to classify the input image, including using a first subset of the set of tiles having highest scores and a second subset of the set of tiles having lowest scores and not using any remaining tiles of the set of tiles as input to a classifier neural network.
18. The non-transitory machine readable medium according to claim 17 , wherein said first convolutional network is a semantic segmentation neural network classifying each pixel of the input image as one of region of interest and a background region.
19. The non-transitory machine readable medium according to claim 17 , wherein the tiling comprises:
applying a fixed tiling grid to at least the region of interest, wherein each of the set of tiles has a predetermined size.
20. The non-transitory machine readable medium according to claim 17 , wherein the tiling further comprises:
augmenting the set of tiles by performing an action on the set of tiles, wherein the action is at least one of a rotation, translation, cropping, adding noise to the input image, modifying an intensity of one or more colors, or changing a contrast of the input image.
21. The non-transitory machine readable medium according to claim 17 , wherein the tiling further comprises:
normalizing a number of tiles in the set of tiles per the input image by performing at least one of a random sampling of the set of tiles and padding the set of tiles with blank tiles, so that said set of tiles comprises a given number of tiles per the input image.
22. The non-transitory machine readable medium according to claim 17 , wherein the method further comprises:
pre-training weights of the second convolutional neural network on at least one of another dataset and a task, wherein at least one output layer is removed from the second convolutional neural network.
23. The non-transitory machine readable medium according to claim 17 , wherein the processing comprises:
for each tile,
computing the score associated with that tile from the extracted feature vector, wherein said tile score represents a contribution of that tile to the classification of the input image;
sorting a set of the tile scores;
selecting the first subset having the highest scores and the second subset having the lowest scores based on at least one of a value and a rank of a tile in the sorted set of tile scores, and
applying the classifier neural network to the subset of tile scores to classify the image.
24. The non-transitory machine readable medium according to claim 17 , wherein the tiling further comprises:
aggregating groups of corresponding tiles from the different ones of a plurality images, wherein the plurality of images includes the input image.
25. The non-transitory machine readable medium according to claim 17 , wherein the tiling further comprises:
aggregating clusters of tiles from the set of tiles that are within a given distance to each other according to a distance metric computed in at least one of the input image or a feature map.
26. The non-transitory machine readable medium according to claim 25 , wherein aggregating a cluster of tiles comprises at least one of:
concatenating tiles in the cluster of tiles;
selecting a single tile from the cluster of tiles based on at least a given criterion, using the cluster of tiles as a multidimensional object for the feature vector extraction;
computing a mean of the feature vectors of tiles in the cluster of tiles; and
computing at least the maximum or minimum value of the feature vectors of tiles in the cluster of tiles.
27. The non-transitory machine readable medium according to claim 17 , wherein the feature vector extraction further comprises:
applying an autoencoder on the extracted feature vectors to reduce the dimensionality of the features.
28. The non-transitory machine readable medium according to claim 17 , wherein the input image is a histopathology slide and said region of interest is a tissue region.
29. The non-transitory machine readable medium according to claim 17 , wherein the image classification is used to perform at least one of a diagnosis classification, survival prediction, and response to treatment prediction.
30. The non-transitory machine readable medium according to claim 17 , wherein the classification of the input image includes predicting at least one type of global and local label.