Systems and methods for specimen interpretation
Systems, methods, devices, and other techniques using machine learning for interpreting, or assisting in the interpretation of, biologic specimens based on digital images are provided. Methods for improving image-based cellular identification, diagnostic methods, methods for evaluating effectiveness of a disease intervention, and visual outputs useful in assisting professionals in the interpretation of biologic specimens are also provided.
1. A method, comprising:
identifying, by a system of one or more computers, a first feature vector that represents cytomorphologic criteria for each of one or more individual cells within a plurality of cells in at least a portion of a whole specimen slide image, wherein the first feature vector includes a plurality of cell-level features;
identifying, by the system, a second feature vector indicative of slide-level features, wherein identifying the second feature vector comprises:
aggregating the plurality of cell-level features; and based on the aggregation, determining a set of metrics representing the whole specimen slide image; and
providing, by the system, the first and the second feature vectors to a machine learning model configured to generate a plurality of feature scores, each feature score corresponding to each feature in the first and the second feature vectors, wherein the feature score is predictive of a presence or absence of a disease or disease type identified in the whole specimen slide image; and
determining, by the system and based on the plurality of feature scores, the presence or absence of a disease or disease type.
2. The method of claim 1 , further comprising, prior to identifying the first feature vector:
receiving, by the system, an image of a whole specimen slide comprising a plurality of biological cells;
detecting, by the system, at least a portion of each of one or more individual cells within the plurality of cells; and
determining, by the system, spatial coordinates for each of the one or more individual cells.
3. The method of claim 2 , further comprising extracting, by the system, for each of the one or more individual cells, an extracted image of the individual cell, wherein the cell is centered on the extracted image, each extracted image representing an independent individual cell.
4. The method of claim 3 , further comprising:
processing, by the system, the one or more extracted images to generate a cell type score for each extracted image; and
extracting, by the system, a set of one or more of the extracted images having a cell type score within a predetermined range, wherein the cell type score a likelihood that the cell is a target cell type.
5. The method of claim 1 , further comprising ranking, by the system, the individual cells based on the plurality of feature scores.
6. The method of claim 1 , further comprising classifying, by the system and using a Gaussian mixture model, each of the individual cells into one of a plurality of predetermined cell-type clusters based on the plurality of feature scores.
7. The method of claim 1 , wherein the machine learning model is a convolutional neural network model.
8. The method of claim 1 , wherein the plurality of cell-level features is selected from the group consisting of a nuclear-to-cytoplasmic ratio, nuclear hyperchromasia, chromatin coarseness, nuclear membrane irregularity, cellular degradation, malignancy classifier, malignancy value, focal score, nuclear-to-cytoplasmic pixel ratio, cell-in-cell arrangements, and combinations thereof.
9. The method of claim 1 , further comprises generating summary statistics based on the first feature vector.
10. The method of claim 9 , wherein the summary statistics are selected from the group consisting of mean, median, standard deviation, variance, kurtosis, or skew, histograms, principal components analysis, and combinations thereof.
11. The method of claim 1 , further comprising:
providing, by the system, one or more outputs indicative of the presence or absence of a disease or disease type identified in the whole specimen slide, wherein the one or more outputs are selected from the group consisting of summary statistics, a cell type cluster score, one or more feature scores, an image of one or more cells, a composite image having a plurality of images of multiple cells, and combinations thereof.
12. The method of claim 4 , further comprising, prior to processing the one or more extracted images, removing, by the system, background noise in each of the extracted images.
13. The method of claim 1 , wherein determining the presence or absence of a disease or disease type comprises:
obtaining the plurality of feature scores from the machine learning model trained on at least one of a group comprising cytomorphologic criteria and histologic criteria.
14. The method of claim 1 , wherein the machine learning model is trained using both cytomorphologic criteria and histologic criteria.
15. The method of claim 1 , wherein the machine learning model is trained using histological criteria when available and cytomorphologic criteria when the histological criteria is not available.
16. The method of claim 1 , wherein the machine learning model is trained by combining a histological test with a cytomorphologic test.
17. The method of claim 16 , wherein the combining of the histological test with the cytomorphologic tests comprises a comparison of a histological confidence value generated by the histologic test with a cytomorphologic confidence value generated by the cytomorphologic test.
18. The method of claim 1 , wherein the disease or disease type comprises high grade urothelial carcinoma, suspicious for high grade urothelial carcinoma, low grade urothelial neoplasia, atypical urothelial cells, and negative for high grade urothelial carcinoma.
19. The method of claim 1 , further comprises evaluating effectiveness of a cancer intervention measure in a subject having or at risk for developing a cancer, wherein the whole specimen slide image is derived from the subject
applying at least one intervention measure that is commensurate with treating or preventing the cancer; and
determining the effectiveness of the intervention measure.
20. The method of claim 1 , further comprising:
displaying an image of a whole specimen slide including biological cells, the image overlaid with a visual representation of a prediction score for each of the one or more individual cells identified in the slide.
21. The method of claim 20 , wherein the prediction score provides a visual indication of an importance score for at least some of the cells based on the respective cell's importance in determining the presence or absence of a disease or disease type.
22. The method of claim 20 , wherein the prediction score provides a visual indication of a point on a severity scale indicative of a severity of the disease or disease type.
23. The method of claim 1 , further comprising:
displaying a single composite displayed image comprising a plurality of selected individual cell images extracted from at least the portion of the whole specimen slide image.