MACHINE LEARNING FRAMEWORK FOR BREAST CANCER HISTOLOGIC GRADING
A machine learning framework for breast cancer histologic grading is described herein. In an example, a method involves accessing a whole slide image of a specimen. The image is processed using a first, second, third, and fourth machine learning process. A first output of the first machine learning process indicates portions of the image predicted to depict tumor cells. A second output of the second machine learning process corresponds to a mitotic count predicted score for a mitotic count depicted in the image, a third output of the third machine learning process corresponds to a nuclear pleomorphism predicted score for nuclear pleomorphism depicted in the image, and a fourth output of the fourth machine learning process corresponds to a tubule formation predicted score for tubule formation depicted in the image. A combined score of a predicted histologic grade of a disease in the image is generated based on the outputs.
1 . A computer-implemented method comprising:
accessing a whole slide image of a specimen, wherein the image comprises a depiction of cells corresponding to a disease;
processing the image using a first machine learning process, wherein a first output of the first machine learning process corresponds to a mask indicating particular portions of the image predicted to depict the tumor cells;
applying the mask to the image to generate a masked image;
processing the masked image using a second machine learning process, wherein a second output of the second machine learning process corresponds to a mitotic count predicted score for a mitotic count depicted in the image;
processing the masked image using a third machine learning process, wherein a third output of the third machine learning process corresponds to a nuclear pleomorphism predicted score for nuclear pleomorphism depicted in the image;
processing the masked image using a fourth machine learning process, wherein a fourth output of the fourth machine learning process corresponds to a tubule formation predicted score for tubule formation depicted in the image;
generating a combined score of a predicted histologic grade of the disease in the image based on the second output, the third output, and the fourth output; and
outputting the combined score of the predicted histologic grade.
2 . The computer-implemented method of claim 1 , wherein the first machine learning process comprises a first machine learning model that segments tumor cells in the image to generate the mask.
3 . The computer-implemented method of claim 1 , wherein the second machine learning process comprises:
generating a first set of patches of the image, wherein each patch of the first set of patches corresponds to a portion of the image;
generating, for each patch of the first set of patches, a mitotic count patch-level score by inputting the patches into a second machine learning model, wherein the mitotic count patch-level score corresponds to a likelihood of the patch corresponding to a mitotic figure;
determining a plurality of metrics corresponding to mitotic density of the image based on the mitotic count patch-level score for each patch of the first set of patches; and
generating the mitotic count predicted score for the image by inputting the plurality of metrics into a third machine learning model.
4 . The computer-implemented method of claim 1 , wherein the third machine learning process comprises:
generating a second set of patches of the image, wherein each patch of the second set of patches corresponds to a portion of the image;
generating, for each patch of the second set of patches, a nuclear pleomorphism patch-level score by inputting the patch into a fourth machine learning model, wherein the nuclear pleomorphism patch-level score corresponds to a likelihood of the patch corresponding to each grade score of a plurality of grade scores associated with nuclear pleomorphism;
determining a metric associated with each grade score of the plurality of grade scores; and
generating the nuclear pleomorphism predicted score for the image by inputting the metric associated with each grade score of the plurality of grade scores into a fifth machine learning model.
5 . The computer-implemented method of claim 1 , wherein the fourth machine learning process comprises:
generating a third set of patches of the image, wherein each patch of the third set of patches corresponds to a portion of the image;
generating, for each patch of the third set of patches, a tubule formation patch-level score by inputting the patch into a sixth machine learning model, wherein the tubule formation patch-level score corresponds to a likelihood of the patch corresponding to each grade score of a plurality of grade scores associated with tubule formation;
determining a metric associated with each grade score of the plurality of grade scores; and
generating the tubule formation predicted score for the image by inputting the metric associated with each grade score of the plurality of grade scores into a seventh machine learning model.
6 . The computer-implemented method of claim 1 , wherein the first machine learning process, the second machine learning process, the third machine learning process, and the fourth machine learning process comprise a convolutional neural network.
7 . The computer-implemented method of claim 1 , wherein the combined score comprises a continuous score between a first value and a second value.
8 . The computer-implemented method of claim 1 , further comprising:
characterizing, classifying, or a combination thereof, the image with respect to the disease based on the combined score; and
outputting, an inference based on the characterizing, classifying, or the combination thereof.
9 . The computer-implemented method of claim 8 , further comprising: determining a diagnosis of a subject associated with the image, wherein the diagnosis is determined based on the inference.
10 . The computer-implemented method of claim 9 , further comprising administering a treatment to the subject based on (i) the inference and/or (ii) the diagnosis of the subject.
11 . A system comprising:
one or more data processors; and
a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform operations comprising:
accessing a whole slide image of a specimen, wherein the image comprises a depiction of cells corresponding to a disease;
processing the image using a first machine learning process, wherein a first output of the first machine learning process corresponds to a mask indicating particular portions of the image predicted to depict the tumor cells;
applying the mask to the image to generate a masked image;
processing the masked image using a second machine learning process, wherein a second output of the second machine learning process corresponds to a mitotic count predicted score for a mitotic count depicted in the image;
processing the masked image using a third machine learning process, wherein a third output of the third machine learning process corresponds to a nuclear pleomorphism predicted score for nuclear pleomorphism depicted in the image;
processing the masked image using a fourth machine learning process, wherein a fourth output of the fourth machine learning process corresponds to a tubule formation predicted score for tubule formation depicted in the image;
generating a combined score of a predicted histologic grade of the disease in the image based on the second output, the third output, and the fourth output; and
outputting the combined score of the predicted histologic grade.
12 . The system of claim 11 , wherein the first machine learning process comprises a first machine learning model that segments tumor cells in the image to generate the mask.
13 . The system of claim 11 , wherein the second machine learning process comprises:
generating a first set of patches of the image, wherein each patch of the first set of patches corresponds to a portion of the image;
generating, for each patch of the first set of patches, a mitotic count patch-level score by inputting the patches into a second machine learning model, wherein the mitotic count patch-level score corresponds to a likelihood of the patch corresponding to a mitotic figure;
determining a plurality of metrics corresponding to mitotic density of the image based on the mitotic count patch-level score for each patch of the first set of patches; and
generating the mitotic count predicted score for the image by inputting the plurality of metrics into a third machine learning model.
14 . The system of claim 11 , wherein the third machine learning process comprises:
generating a second set of patches of the image, wherein each patch of the second set of patches corresponds to a portion of the image;
generating, for each patch of the second set of patches, a nuclear pleomorphism patch-level score by inputting the patch into a fourth machine learning model, wherein the nuclear pleomorphism patch-level score corresponds to a likelihood of the patch corresponding to each grade score of a plurality of grade scores associated with nuclear pleomorphism;
determining a metric associated with each grade score of the plurality of grade scores; and
generating the nuclear pleomorphism predicted score for the image by inputting the metric associated with each grade score of the plurality of grade scores into a fifth machine learning model.
15 . The system of claim 1 , wherein the fourth machine learning process comprises:
generating a third set of patches of the image, wherein each patch of the third set of patches corresponds to a portion of the image;
generating, for each patch of the third set of patches, a tubule formation patch-level score by inputting the patch into a sixth machine learning model, wherein the tubule formation patch-level score corresponds to a likelihood of the patch corresponding to each grade score of a plurality of grade scores associated with tubule formation;
determining a metric associated with each grade score of the plurality of grade scores; and
generating the tubule formation predicted score for the image by inputting the metric associated with each grade score of the plurality of grade scores into a seventh machine learning model.
16 . The system of claim 11 , wherein the first machine learning process, the second machine learning process, the third machine learning process, and the fourth machine learning process comprise a convolutional neural network.
17 . The system of claim 11 , wherein the combined score comprises a continuous score between a first value and a second value.
18 . The system of claim 11 , wherein the operations further comprise:
characterizing, classifying, or a combination thereof, the image with respect to the disease based on the combined score; and
outputting, an inference based on the characterizing, classifying, or the combination thereof.
19 . The system of claim 18 , wherein the operations further comprise: determining a diagnosis of a subject associated with the image, wherein the diagnosis is determined based on the inference.
20 . (canceled)
21 . A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform operations comprising:
accessing a whole slide image of a specimen, wherein the image comprises a depiction of cells corresponding to a disease;
processing the image using a first machine learning process, wherein a first output of the first machine learning process corresponds to a mask indicating particular portions of the image predicted to depict the tumor cells;
applying the mask to the image to generate a masked image;
processing the masked image using a second machine learning process, wherein a second output of the second machine learning process corresponds to a mitotic count predicted score for a mitotic count depicted in the image;
processing the masked image using a third machine learning process, wherein a third output of the third machine learning process corresponds to a nuclear pleomorphism predicted score for nuclear pleomorphism depicted in the image;
processing the masked image using a fourth machine learning process, wherein a fourth output of the fourth machine learning process corresponds to a tubule formation predicted score for tubule formation depicted in the image;
generating a combined score of a predicted histologic grade of the disease in the image based on the second output, the third output, and the fourth output; and
outputting the combined score of the predicted histologic grade.
22 .- 30 . (canceled)