Method and system for joint multi-organ segmentation in medical image data using local and global context

1. A method for segmenting a plurality of organs in a medical image, comprising:

detecting a plurality of landmarks of the plurality of organs in the medical image using an integrated local and global context detector, wherein detecting a plurality of landmarks of the plurality of organs in the medical image using an integrated local and global context detector comprises:

generating a respective set of location predictions for each of the plurality of landmarks from each of a plurality of image patches of the medical image using a global context posterior, and

for each of the plurality of landmarks, detecting the landmark by evaluating the predicted locations for that landmark based on a local context using a trained discriminative classifier for that landmark; and

generating a segmentation of each of the plurality of organs based on the detected plurality of landmarks.

2. The method of claim 1 , wherein generating a respective set of location predictions for each of the plurality of landmarks from each of a plurality of image patches of the medical image using a global context posterior comprises, for each image patch:

determining a number of nearest neighbor image patches in a database of image patches;

determining location predictions for each of the plurality of landmarks based on a relative shift vector corresponding to each of the nearest neighbor image patches, wherein each relative shift vector includes relative shifts between a voxel in a training volume at which the corresponding nearest neighbor image patch is centered and each of the plurality of landmarks in the training volume.

3. The method of claim 2 , wherein determining location predictions for each of the plurality of landmarks based on a relative shift vector corresponding to each of the nearest neighbor image patches comprises:

generating a prediction for each of the plurality of landmarks from each relative shift vector by applying the relative shift associated with each respective landmark to a voxel in the medical image at which the image patch is centered.

4. The method of claim 1 , wherein generating a respective set of location predictions for each of the plurality of landmarks from each of a plurality of image patches of the medical image using a global context posterior comprises:

for each of the plurality of image patches, simultaneously generating the location predictions for each of the plurality of landmarks using the global context posterior.

5. The method of claim 1 , wherein generating a respective set of location predictions for each of the plurality of landmarks from each of a plurality of image patches of the medical image using a global context posterior comprises:

generating a subset of voxels sampled from a total number of voxels in the medical image; and

generating the location predictions for each of the plurality of landmarks from a respective image patch centered at each of the subset of voxels using the global context posterior.

6. The method of claim 1 , wherein generating a segmentation of each of the plurality of organs based on the detected plurality of landmarks comprises:

aligning a respective trained statistical shape model for each organ to the medical image based on a respective subset of the plurality of landmarks for each organ.

7. The method of claim 1 , wherein generating a segmentation of each of the plurality of organs based on the detected plurality of landmarks comprises:

aligning a trained statistical shape model of a joint shape of the plurality of organs to the medical image based on the plurality of landmarks.

8. The method of claim 1 , further comprising:

refining a boundary of the segmentation for each of the plurality of organs using a trained boundary detector for each organ.

9. The method of claim 8 , wherein refining a boundary of the segmentation for each of the plurality of organs using a trained boundary detector for each organ comprises, for each organ:

iteratively displacing each vertex on a surface of a mesh segmented for the organ along a surface normal to maximize a response of the trained boundary detector, resulting in an adjusted mesh; and

projecting the adjusted mesh onto a linear subspace spanned by a linear shape model of the organ.

10. An apparatus for segmenting a plurality of organs in a medical image, comprising:

means for detecting a plurality of landmarks of the plurality of organs in the medical image using an integrated local and global context detector,

wherein the means for detecting a plurality of landmarks of the plurality of organs in the medical image using an integrated local and global context detector comprises:

means for generating a respective set of location predictions for each of the plurality of landmarks from each of a plurality of image patches of the medical image using a global context posterior, and

means for detecting each of the plurality of landmarks by evaluating the predicted locations for that landmark based on a local context using a trained discriminative classifier for that landmark; and

means for generating a segmentation of each of the plurality of organs based on the detected plurality of landmarks.

11. The apparatus of claim 10 , wherein the means for generating a respective set of location predictions for each of the plurality of landmarks from each of a plurality of image patches of the medical image using a global context posterior comprises:

means for determining, for each image patch, a number of nearest neighbor image patches in a database of image patches;

means for determining location predictions for each of the plurality of landmarks based on a relative shift vector corresponding to each of the nearest neighbor image patches, wherein each relative shift vector includes relative shifts between a voxel in a training volume at which the corresponding nearest neighbor image patch is centered and each of the plurality of landmarks in the training volume.

12. The apparatus of claim 11 , wherein the means for determining location predictions for each of the plurality of landmarks based on a relative shift vector corresponding to each of the nearest neighbor image patches comprises:

means for generating a prediction for each of the plurality of landmarks from each relative shift vector by applying the relative shift associated with each respective landmark to a voxel in the medical image at which the image patch is centered.

13. The apparatus of claim 10 , wherein the means for generating a respective set of location predictions for each of the plurality of landmarks from each of a plurality of image patches of the medical image using a global context posterior comprises:

means for simultaneously generating the location predictions for each of the plurality of landmarks from an image patch using the global context posterior.

14. The apparatus of claim 10 , wherein the means for generating a respective set of location predictions for each of the plurality of landmarks from each of a plurality of image patches of the medical image using a global context posterior comprises:

means for generating a subset of voxels sampled from a total number of voxels in the medical image; and

means for generating the location predictions for each of the plurality of landmarks from a respective image patch centered at each of the subset of voxels using the global context posterior.

15. The apparatus of claim 10 , wherein the means for generating a segmentation of each of the plurality of organs based on the detected plurality of landmarks comprises:

means for aligning a respective trained statistical shape model for each organ to the medical image based on a respective subset of the plurality of landmarks for each organ.

16. The apparatus of claim 10 , wherein the means for generating a segmentation of each of the plurality of organs based on the detected plurality of landmarks comprises:

means for aligning a trained statistical shape model of a joint shape of the plurality of organs to the medical image based on the plurality of landmarks.

17. The apparatus of claim 10 , further comprising:

means for refining a boundary of the segmentation for each of the plurality of organs using a trained boundary detector for each organ.

18. A non-transitory computer readable medium storing computer program instructions for segmenting a plurality of organs in a medical image, the computer program instructions when executed on a processor cause the processor to perform operations comprising:

detecting a plurality of landmarks of the plurality of organs in the medical image using an integrated local and global context detector, wherein detecting a plurality of landmarks of the plurality of organs in the medical image using an integrated local and global context detector comprises:

generating a respective set of location predictions for each of the plurality of landmarks from each of a plurality of image patches of the medical image using a global context posterior, and

for each of the plurality of landmarks, detecting the landmark by evaluating the predicted locations for that landmark based on a local context using a trained discriminative classifier for that landmark; and

generating a segmentation of each of the plurality of organs based on the detected plurality of landmarks.

19. The non-transitory computer readable medium of claim 18 , wherein generating a respective set of location predictions for each of the plurality of landmarks from each of a plurality of image patches of the medical image using a global context posterior comprises, for each image patch:

determining a number of nearest neighbor image patches in a database of image patches;

determining location predictions for each of the plurality of landmarks based on a relative shift vector corresponding to each of the nearest neighbor image patches, wherein each relative shift vector includes relative shifts between a voxel in a training volume at which the corresponding nearest neighbor image patch is centered and each of the plurality of landmarks in the training volume.

20. The non-transitory computer readable medium of claim 19 , wherein determining location predictions for each of the plurality of landmarks based on a relative shift vector corresponding to each of the nearest neighbor image patches comprises:

generating a prediction for each of the plurality of landmarks from each relative shift vector by applying the relative shift associated with each respective landmark to a voxel in the medical image at which the image patch is centered.

21. The non-transitory computer readable medium of claim 18 , wherein generating a respective set of location predictions for each of the plurality of landmarks from each of a plurality of image patches of the medical image using a global context posterior comprises:

for each of the plurality of image patches, simultaneously generating the location predictions for each of the plurality of landmarks using the global context posterior.

22. The non-transitory computer readable medium of claim 18 , wherein generating a respective set of location predictions for each of the plurality of landmarks from each of a plurality of image patches of the medical image using a global context posterior comprises:

generating a subset of voxels sampled from a total number of voxels in the medical image; and

generating the location predictions for each of the plurality of landmarks from a respective image patch centered at each of the subset of voxels using the global context posterior.

23. The non-transitory computer readable medium of claim 18 , wherein generating a segmentation of each of the plurality of organs based on the detected plurality of landmarks comprises:

aligning a respective trained statistical shape model for each organ to the medical image based on a respective subset of the plurality of landmarks for each organ.

24. The non-transitory computer readable medium of claim 18 , wherein generating a segmentation of each of the plurality of organs based on the detected plurality of landmarks comprises:

aligning a trained statistical shape model of a joint shape of the plurality of organs to the medical image based on the plurality of landmarks.

25. The non-transitory computer readable medium of claim 18 , the operations further comprising:

refining a boundary of the segmentation for each of the plurality of organs using a trained boundary detector for each organ.

26. The non-transitory computer readable medium of claim 25 , wherein refining a boundary of the segmentation for each of the plurality of organs using a trained boundary detector for each organ comprises, for each organ:

iteratively displacing each vertex on a surface of a mesh segmented for the organ along a surface normal to maximize a response of the trained boundary detector, resulting in an adjusted mesh; and

projecting the adjusted mesh onto a linear subspace spanned by a linear shape model of the organ.