Systems and methods to process electronic images to provide localized semantic analysis of whole slide images
Systems and methods are disclosed for identifying formerly conjoined pieces of tissue in a specimen, comprising receiving one or more digital images associated with a pathology specimen, identifying a plurality of pieces of tissue by applying an instance segmentation system to the one or more digital images, the instance segmentation system having been generated by processing a plurality of training images, determining, using the instance segmentation system, a prediction of whether any of the plurality of pieces of tissue were formerly conjoined, and outputting at least one instance segmentation to a digital storage device and/or display, the instance segmentation comprising an indication of whether any of the plurality of pieces of tissue were formerly conjoined.
1. A computer-implemented method of training a machine learning model to identify previously conjoined pieces of tissue in a specimen using a synthetic dataset, comprising:
receiving a plurality of digital images associated with a plurality of pathology specimens, at least a portion of the plurality of digital images comprising images of previously conjoined pieces of tissue;
generating a synthetic dataset based on the plurality of digital images;
training a panoptic or instance segmentation model based on the plurality of digital images and the synthetic dataset; and
outputting the panoptic or instance segmentation model to at least one digital storage device, wherein generating the synthetic dataset further comprises:
generating a bank of tissue cores based on the plurality of digital images, the plurality of digital images including one or more annotated digital images;
generating an empty synthetic slide by sampling background or selecting a fixed background color from the one or more annotated digital images;
randomly placing and/or rotating one or more tissue cores and corresponding tissue mask from the bank of tissue cores onto the empty synthetic slide; and
placing the empty synthetic slide with one or more tissue cores and corresponding tissue mask on an array.
2. The computer-implemented method of claim 1 , wherein generating the synthetic dataset further comprises:
generating at least one slide background for images associated with the synthetic dataset;
embedding one or more artifact and/or floater for at least one of the images associated with the synthetic dataset; and
embedding one or more levels and/or cores for at least one of the images associated with the synthetic dataset.
3. The computer-implemented method of claim 2 , wherein the synthetic dataset comprises at least one annotation.
4. The computer-implemented method of claim 1 , wherein each annotated digital image comprises at least one annotation, and each annotation takes a form of a polygon that segments a distinct region of tissue of the pathology specimen, and
the method further comprising:
determining a tissue mask based on each polygon, each tissue mask segmenting tissue from a background;
converting the corresponding tissue mask into a single annotation mask for an entirety of the empty synthetic slide to generate a synthetic digital image;
outputting the synthetic digital image and the at least one annotation; and
saving the synthetic digital image and the at least one annotation to a digital storage device.
5. The computer-implemented method of claim 1 , further comprising adding random noise to the array.
6. The computer-implemented method of claim 1 , further comprising:
running the panoptic or instance segmentation model on a region of interest in a pathology specimen to generate a report; and
outputting the report to a digital storage device and/or display.
7. The computer-implemented method of claim 6 , wherein the report comprises at least one of an identification of a specimen type, an association of two or more tissue fragments that belong together, and/or a correspondence of two or more sectioned levels of the pathology specimen.
8. The computer-implemented method of claim 1 , further comprising:
running the panoptic or instance segmentation model on one or more digital pathology slides to generate at least one of one or more labeled tissue cores and/or levels, one or more artifacts, and/or a background; and
determining whether any tissue regions of the digital pathology slides belong to a similar core at a different level.
9. The computer-implemented method of claim 1 , further comprising:
running the panoptic or instance segmentation model on one or more digital pathology slides to identify one or more tissue regions across the one or more digital pathology slides;
matching the one or more tissue regions as belonging to a similar tissue block at a different level; and
outputting at least one matching tissue regions to a digital storage.
10. The computer-implemented method of claim 9 , where matching the one or more tissues regions comprises using a correlation-based method.
11. The computer-implemented method of claim 9 , where matching the one or more tissue regions comprises using a feature-based method.
12. A system for training a machine learning model to identify previously conjoined pieces of tissue in a specimen using a synthetic dataset, the system comprising:
at least one memory storing instructions; and
at least one processor configured to execute the instructions to perform operations comprising:
receiving a plurality of digital images associated with a plurality of pathology specimens, at least a portion of the plurality of digital images comprising images of previously conjoined pieces of tissue;
generating a synthetic dataset based on the plurality of digital images;
training a panoptic or instance segmentation model based on the plurality of digital images and the synthetic dataset; and
outputting the panoptic or instance segmentation model to at least one digital storage device, wherein generating the synthetic dataset further comprises:
generating a bank of tissue cores based on the plurality of digital images, the plurality of digital images including one or more annotated digital images;
generating an empty synthetic slide by sampling background or selecting a fixed background color from the one or more annotated digital images;
randomly placing and/or rotating one or more tissue cores and corresponding tissue mask from the bank of tissue cores onto the empty synthetic slide; and
placing the empty synthetic slide with one or more tissue cores and corresponding tissue mask on an array.
13. The system of claim 12 , wherein generating the synthetic dataset further comprises:
generating at least one slide background for images associated with the synthetic dataset;
embedding one or more artifact and/or floater for at least one of the images associated with the synthetic dataset; and
embedding one or more levels and/or cores for at least one of the images associated with the synthetic dataset.
14. The system of claim 12 , wherein each annotated digital image comprises at least one annotation, and each annotation takes a form of a polygon that segments a distinct region of tissue of the pathology specimen, and
wherein the operations further comprise:
determining a tissue mask based on each polygon, each tissue mask segmenting tissue from a background;
converting the corresponding tissue mask into a single annotation mask for an entirety of the empty synthetic slide to generate a synthetic digital image;
outputting the synthetic digital image and the at least one annotation; and
saving the synthetic digital image and the at least one annotation to a digital storage device.
15. The system of claim 12 , wherein the operations further comprise:
running the panoptic or instance segmentation model on a region of interest in a pathology specimen to generate a report; and
outputting the report to a digital storage device and/or display.
16. The system of claim 15 , wherein the report comprises at least one of an identification of a specimen type, an association of two or more tissue fragments that belong together, and/or a correspondence of two or more sectioned levels of the pathology specimen.
17. The system of claim 12 , wherein the operations further comprise:
running the panoptic or instance segmentation model on one or more digital pathology slides to identify one or more tissue regions across the one or more digital pathology slides;
matching the one or more tissue regions as belonging to a similar tissue block at a different level; and
outputting at least one matching tissue regions to a digital storage.
18. The system of claim 17 , where matching the one or more tissues regions comprises using a correlation-based method.
19. A non-transitory machine-readable medium storing instructions that, when executed by a computing system, causes the computing system to perform a method of training a machine learning model to identify previously conjoined pieces of tissue in a specimen using a synthetic dataset, the method including:
receiving a plurality of digital images associated with a plurality of pathology specimens, at least a portion of the plurality of digital images comprising images of previously conjoined pieces of tissue;
generating a synthetic dataset based on the plurality of digital images;
training a panoptic or instance segmentation model based on the plurality of digital images and the synthetic dataset; and
outputting the panoptic or instance segmentation model to at least one digital storage device, wherein generating the synthetic dataset further comprises:
generating a bank of tissue cores based on the plurality of digital images, the plurality of digital images including one or more annotated digital images;
generating an empty synthetic slide by sampling background or selecting a fixed background color from the one or more annotated digital images;
randomly placing and/or rotating one or more tissue cores and corresponding tissue mask from the bank of tissue cores onto the empty synthetic slide; and
placing the empty synthetic slide with one or more tissue cores and corresponding tissue mask on an array.
20. The non-transitory machine-readable medium of claim 19 , wherein each annotated digital image comprises at least one annotation, and each annotation takes a form of a polygon that segments a distinct region of tissue of the pathology specimen, and
wherein the method further comprises:
determining a tissue mask based on each polygon, each tissue mask segmenting tissue from a background;
converting the corresponding tissue mask into a single annotation mask for an entirety of the empty synthetic slide to generate a synthetic digital image;
outputting the synthetic digital image and the at least one annotation; and
saving the synthetic digital image and the at least one annotation to a digital storage device.