System and method for multi-scale coarse-to-fine segmentation of images to detect pancreatic ductal adenocarcinoma

1. A data processing apparatus for processing images through a system of one or more neural networks to detect a pancreatic ductal adenocarcinoma (PDAC), the data processing apparatus including one or more processors and one or more computer readable storage media storing instructions, that when executed by the one or more processors, causes the one or more processors to perform operations, the operations comprising:

obtaining a first image;

performing coarse segmentation of the first image using a first neural network trained to produce first output data representing a first likelihood that the first image includes more than a threshold number of voxels that depict a tumor;

determining a region of interest of the first image based on the coarse segmentation;

performing multi-stage fine segmentation using a plurality of images that are each generated based on the region of interest of the first image, comprising:

performing a first fine stage segmentation of a second image generated based on the region of interest of the first image using a second neural network trained to produce second output data representing a second likelihood that the second image includes more than a threshold number of voxels that depict a tumor; and

performing a second fine stage segmentation of a third image generated based on the region of interest of the first image using a third neural network trained to produce third output data representing a third likelihood that the third image includes more than a threshold number of voxels that depict a tumor; and

determining, based on the first output data, the second output data, and the third output data whether the obtained first image depicts a tumor.

2. The data processing apparatus of claim 1 , wherein the first image, the second image, and the third image are each CT images.

3. The data processing apparatus of claim 1 , wherein determining based on the first output data, the second output data, and the third output data whether the first image depicts a tumor comprises:

determining average data that represents an average of the first likelihood, the second likelihood, and the third likelihood; and

determining whether the average data satisfies a predetermined threshold.

4. The data processing apparatus of claim 3 , wherein the operations further comprise:

based on determining that the average data satisfies the predetermined threshold, determining that the first image depicts a tumor.

5. The data processing apparatus of claim 3 , wherein the operations further comprise:

based on determining that the average data does not satisfy the predetermined threshold, determining that the first image does not depict a tumor.

6. The data processing apparatus of claim 1 , wherein determining, based on the first output data, the second output data, and the third output data whether the obtained first image depicts a tumor comprises:

classifying each voxel in the first image as either normal or abnormal based on the first output data, the second output data, and the third output data; and

generating a graph structure that represents each voxel in the first image as a node with an edge between each adjacent voxel.

7. The data processing apparatus of claim 6 , the operations further comprising:

identifying sets of connected voxels based on the nodes and edges of the graph structure; and

determining that a particular set of the identified sets of connected voxels is likely to be a tumor if the particular set of connected voxels includes more than a threshold number of predicted tumor voxels.

8. A method comprising:

obtaining a first image;

performing coarse segmentation of the first image using a first neural network trained to produce first output data representing a first likelihood that the first image includes more than a threshold number of voxels that depict a tumor;

determining a region of interest of the first image based on the coarse segmentation;

performing multi-stage fine segmentation using a plurality of images that are each generated based on the region of interest of the first image, comprising:

performing a first fine stage segmentation of a second image generated based on the region of interest of the first image using a second neural network trained to produce second output data representing a second likelihood that the second image includes more than a threshold number of voxels that depict a tumor; and

performing a second fine stage segmentation of a third image generated based on the region of interest of the first image using a third neural network trained to produce third output data representing a third likelihood that the third image includes more than a threshold number of voxels that depict a tumor; and

determining, based on the first output data, the second output data, and the third output data whether the obtained first image depicts a tumor.

9. The method of claim 8 , wherein determining based on the first output data, the second output data, and the third output data whether the first image depicts a tumor comprises:

determining average data that represents an average of the first likelihood, the second likelihood, and the third likelihood; and

determining whether the average data satisfies a predetermined threshold.

10. The method of claim 9 , wherein the method further comprises:

based on determining that the average data satisfies the predetermined threshold, determining that the first image depicts a tumor.

11. The method of claim 9 , wherein the method further comprises:

based on determining that the average data does not satisfy the predetermined threshold, determining that the first image does not depict a tumor.

12. The method of claim 8 , wherein determining, based on the first output data, the second output data, and the third output data whether the obtained first image depicts a tumor comprises:

classifying each voxel in the first image as either normal or abnormal based on the first output data, the second output data, and the third output data; and

generating a graph structure that represents each voxel in the first image as a node with an edge between each adjacent voxel.

13. The method of claim 12 , the method further comprises:

identifying sets of connected voxels based on the nodes and edges of the graph structure; and

determining that a particular set of the identified sets of connected voxels is likely to be a tumor if the particular set of connected voxels includes more than a threshold number of predicted tumor voxels.

14. A computer-readable storage device having stored thereon instructions, which, when executed by data processing apparatus, cause the data processing apparatus to perform operations, the operations comprising:

obtaining a first image;

performing coarse segmentation of the first image using a first neural network trained to produce first output data representing a first likelihood that the first image includes more than a threshold number of voxels that depict a tumor;

determining a region of interest of the first image based on the coarse segmentation;

performing multi-stage fine segmentation using a plurality of images that are each generated based on the region of interest of the first image, comprising:

performing a first fine stage segmentation of a second image generated based on the region of interest of the first image using a second neural network trained to produce second output data representing a second likelihood that the second image includes more than a threshold number of voxels that depict a tumor; and

performing a second fine stage segmentation of a third image generated based on the region of interest of the first image using a third neural network trained to produce third output data representing a third likelihood that the third image includes more than a threshold number of voxels that depict a tumor; and

determining, based on the first output data, the second output data, and the third output data whether the obtained first image depicts a tumor.

15. The computer-readable storage device of claim 14 , wherein determining based on the first output data, the second output data, and the third output data whether the first image depicts a tumor comprises:

determining average data that represents an average of the first likelihood, the second likelihood, and the third likelihood; and

determining whether the average data satisfies a predetermined threshold.

16. The computer-readable storage device of claim 15 , wherein the operations further comprise:

based on determining that the average data satisfies the predetermined threshold, determining that the first image depicts a tumor, or

based on determining that the average data does not satisfy the predetermined threshold, determining that the first image does not depict a tumor.

17. The computer-readable storage device of claim 14 , wherein determining, based on the first output data, the second output data, and the third output data whether the obtained first image depicts a tumor comprises:

classifying each voxel in the first image as either normal or abnormal based on the first output data, the second output data, and the third output data; and

generating a graph structure that represents each voxel in the first image as a node with an edge between each adjacent voxel.

18. The computer-readable storage device of claim 17 , the operations further comprising:

identifying sets of connected voxels based on the nodes and edges of the graph structure; and

determining that a particular set of the identified sets of connected voxels is likely to be a tumor if the particular set of connected voxels includes more than a threshold number of predicted tumor voxels.