Weakly supervised learning with whole slide images
Techniques are provided for determining classifications based on WSIs. A varied-size feature map is generated for each training WSI by generating a grid of patches for the training WSI, segmenting the training WSI into tissue and non-tissue areas, and converting patches comprising the tissue areas into tensors. Bounding boxes are generated based on the patches comprising tissue areas and segmented into feature map patches. A fixed-size feature map is generated based on a subset of the feature map patches. A classifier model is trained to process fixed-size feature maps corresponding to the training WSIs such that, for each fixed-size feature map, the classifier model is operable to assign a WSI-level tissue or cell morphology classification or regression based on the tensors. A classification engine is configured to use the trained classifier model to determine a WSI-level tissue or cell morphology classification or regression for a test WSI.
1. A computerized method of determining classifications based on whole slide images (WSIs), comprising:
obtaining a plurality of training WSIs;
generating a varied-size feature map for each of the plurality of training WSIs by generating a grid of patches for the training WSI, segmenting the training WSI into tissue and non-tissue areas, and converting patches comprising the tissue areas into tensors;
generating at least one bounding box based on the patches comprising the tissue areas;
segmenting the at least one bounding box into feature map patches;
generating a fixed-size feature map based on at least a subset of the feature map patches;
configuring a classifier model to process fixed-size feature maps corresponding to the training WSIs such that, for each fixed-size feature map, the classifier model is operable to assign a WSI-level tissue or cell morphology classification or regression based on the tensors;
training the classifier model using the fixed-size feature map corresponding to the plurality of training WSIs; and
configuring a classification engine to use the trained classifier model to determine a WSI-level tissue or cell morphology classification or regression for a test WSI.
2. The method of claim 1 , wherein the plurality of training WSIs comprises less than 1000 pathology slide images.
3. The method of claim 1 , wherein the plurality of training WSIs comprises hematoxylin and eosin (H&E)-stained whole slide images.
4. The method of claim 1 , wherein each of the plurality of training WSIs corresponds to an WSI-level label indicating a classification comprising at least one of a type of cancer and a cancer grade.
5. The method of claim 1 , wherein each of the plurality of training WSIs corresponds to an WSI-level label indicating a regression comprising at least one of a percentage of tumor-infiltrating lymphocytes, RNA expression, mutation burden, and allele frequency.
6. The method of claim 1 , further comprising converting patches of the grid of patches determined to comprise non-tissue areas into tensors comprising white feature components.
7. The method of claim 1 , further comprising:
filtering the grid of patches for a minimum color variance; and
eliminating each patch determined to be empty space or background from further processing based on the filtering.
8. The method of claim 1 , wherein each of the tensors comprises a multidimensional descriptive vector.
9. The method of claim 8 , wherein the multidimensional descriptive vector comprises an RGB component.
10. The method of claim 1 , further comprising converting the RGB component into a feature vector.
11. The method of claim 10 , wherein the feature vector is a 512-feature vector for a resnet34 deep-learning neural network.
12. The method of claim 1 , wherein each of the feature map patches comprises a fixed-size patch.
13. The method of claim 1 , wherein the feature map patches comprise one of (16, 16, N) or (32, 32, N) tensors, and wherein N is a feature vector size.
14. The method of claim 1 , further comprising generating the fixed-size feature map based on a randomly selected subset of the feature map patches.
15. The method of claim 1 , wherein the subset of the feature map patches is arranged randomly within the fixed-size feature map.
16. The method of claim 1 , further comprising selecting the subset of the feature map patches for further processing.
17. The method of claim 16 , wherein the subset of the feature map patches is randomly selected to define cancer-enriched areas.
18. The method of claim 16 , wherein the subset of the feature map patches is selected to summarize tumor content within a training WSI.
19. The method of claim 1 , wherein the fixed-size feature map comprises one of a (256, 256, N) or (224, 224, N) feature map.
20. The method of claim 1 , wherein the classifier model comprises a modified resnet34 deep-learning neural network.
21. The method of claim 1 , wherein the classifier model comprises a two-layer convolutional deep-learning neural network.
22. The method of claim 1 , wherein the classifier model comprises at least one of an Inception-v3, resnet34, resnet152, densenet169, densenet201 or other deep-learning neural network.
23. The method of claim 1 , wherein each of the plurality of training WSIs corresponds to a different patient.
24. The method of claim 1 , further comprising:
obtaining the test WSI;
generating a varied-size feature map for the test WSIs by generating a grid of patches for the test WSI, segmenting the test WSI into tissue and non-tissue areas, and converting patches comprising the tissue areas into tensors;
generating at least one bounding box based on the patches comprising the tissue areas;
segmenting the at least one bounding box into feature map patches;
generating a fixed-size feature map based on at least a subset of the feature map patches; and
processing the fixed-size feature map using the trained classifier model, wherein the trained classifier model is operable to determine a WSI-level tissue or cell morphology classification or regression for the test WSI based on the fixed-size feature map.
25. An apparatus for determining classifications based on whole slide images (WSIs), the apparatus comprising:
a processor;
a memory device storing software instructions for determining molecular subtype classifications; and
a training engine executable on the processor according to software instructions stored in the memory device and configured to:
obtain a plurality of training WSIs;
generate a varied-size feature map for each of the plurality of training WSIs by generating a grid of patches for the training WSI, segmenting the training WSI into tissue and non-tissue areas, and converting patches comprising the tissue areas into tensors;
generate at least one bounding box based on the patches comprising the tissue areas;
segment the at least one bounding box into feature map patches;
generate a fixed-size feature map based on at least a subset of the feature map patches;
configure a classifier model to process fixed-size feature maps corresponding to the training WSIs such that, for each fixed-size feature map, the classifier model is operable to assign a WSI-level tissue or cell morphology classification or regression based on the tensors;
train the classifier model using the fixed-size feature map corresponding to the plurality of training WSIs; and
configure a classification engine to use the trained classifier model to determine a WSI-level tissue or cell morphology classification or regression for a test WSI.
26. A non-transitory computer-readable medium having computer instructions stored thereon for determining classifications based on whole slide images (WSIs), which, when executed by a processor, cause the processor to perform one or more steps comprising:
obtaining a plurality of training WSIs;
generating a varied-size feature map for each of the plurality of training WSIs by generating a grid of patches for the training WSI, segmenting the training WSI into tissue and non-tissue areas, and converting patches comprising the tissue areas into tensors;
generating at least one bounding box based on the patches comprising the tissue areas;
segmenting the at least one bounding box into feature map patches;
generating a fixed-size feature map based on at least a subset of the feature map patches;
configuring a classifier model to process fixed-size feature maps corresponding to the training WSIs such that, for each fixed-size feature map, the classifier model is operable to assign a WSI-level tissue or cell morphology classification or regression based on the tensors;
training the classifier model using the fixed-size feature map corresponding to the plurality of training WSIs; and
configuring a classification engine to use the trained classifier model to determine a WSI-level tissue or cell morphology classification or regression for a test WSI.
27. An apparatus for determining classifications based on whole slide images (WSIs), the apparatus comprising:
a processor;
a memory device storing software instructions for determining tissue or cell morphology classifications or regressions; and
a classification engine executable on the processor according to software instructions stored in the memory device and configured to:
obtain a test WSI;
generate a varied-size feature map for the test WSI by generating a grid of patches for the test WSI, segmenting the test WSI into tissue and non-tissue areas, and converting patches comprising the tissue areas into tensors;
generate at least one bounding box based on the patches comprising the tissue areas;
segment the at least one bounding box into feature map patches;
generate a fixed-size feature map based on at least a subset of the feature map patches; and
determine a WSI-level tissue or cell morphology classification or regression for the test WSI using a classifier model trained to process the fixed-size feature map such that a WSI-level tissue or cell morphology classification or regression is determined based on the fixed-size feature map.
28. The apparatus of claim 27 , wherein the classification engine comprises at least one of a cellular smartphone, kiosk, personal data assistant, tablet, robot, vehicle, web camera, or computing device.