3D depth reconstruction of vessels in 2D medical images

1. A method comprising:

receiving a medical image comprising branches of one or more vessels;

generating a branch overlap image channel representing a pixelwise probability that the branches overlap;

generating a set of branch orientation image channels each associated with one of a plurality of orientations, each branch orientation image channel representing a pixelwise probability that the branches are oriented in its associated orientation; and

generating a multi-channel depth image based on the branch overlap image channel and the set of branch orientation image channels, each channel of the multi-channel depth image comprising portions of the branches corresponding to a respective depth.

2. The method of claim 1 , wherein:

generating a branch overlap image channel representing a pixelwise probability that the branches overlap comprises generating the branch overlap image channel using a first trained machine learning network;

generating a set of branch orientation image channels each associated with one of a plurality of orientations comprises generating the set of branch orientation image channels using a second trained machine learning network; and

generating a multi-channel depth image based on the branch overlap image channel and the set of branch orientation image channels comprises generating the multi-channel depth image using a third trained machine learning network.

3. The method of claim 2 , wherein the first trained machine learning network comprises a trained image to image network, the second trained machine learning network comprises a trained fully convolutional neural network, and the third trained machine learning network comprises another trained image to image network.

4. The method of claim 1 , wherein the plurality of orientations comprises a vertical orientation, a horizontal orientation, a first diagonal orientation, and a second diagonal orientation.

5. The method of claim 1 , wherein generating a set of branch orientation image channels each associated with one of a plurality of orientations comprises:

extracting a plurality of patches from the medical image;

for each particular patch of the plurality of patches:

applying a trained fully convolutional neural network to the particular patch to determine a probability, for each respective orientation of the plurality of orientations, that a pixel associated with the particular patch depicts the branches in the respective orientation; and

for each respective orientation, combining the probability for each pixel associated with each particular patch for the respective orientation to generate the set of branch orientation image channels.

6. The method of claim 1 , further comprising:

highlighting pixels in the branch overlap image channel based on the pixelwise probability that the branches overlap; and

highlighting pixels in the set of branch orientation image channels based on the pixelwise probability that the branches are oriented in their associated orientation.

7. The method of claim 1 , wherein generating a multi-channel depth image based on the branch overlap image channel and the set of branch orientation image channels comprises:

generating the multi-channel depth image based on coded high level features previously generated from another branch overlap image channel and another set of branch orientation image channels, the other branch overlap image channel and the other set of branch orientation image channels being generated from another medical image.

8. The method of claim 7 , wherein generating the multi-channel depth image based on coded high level features previously generated from another branch overlap image channel and another set of branch orientation image channels comprises:

generating the multi-channel depth image using a long short-term memory network.

9. The method of claim 1 , wherein generating a multi-channel depth image based on the branch overlap image channel and the set of branch orientation image channels comprises:

generating the multi-channel depth image based on pixel intensity values of the medical image.

10. The method of claim 1 , further comprising:

performing an imaging analysis task based on the multi-channel depth image.

11. An apparatus comprising:

means for receiving a medical image comprising branches of one or more vessels;

means for generating a branch overlap image channel representing a pixelwise probability that the branches overlap;

means for generating a set of branch orientation image channels each associated with one of a plurality of orientations, each branch orientation image channel representing a pixelwise probability that the branches are oriented in its associated orientation; and

means for generating a multi-channel depth image based on the branch overlap image channel and the set of branch orientation image channels, each channel of the multi-channel depth image comprising portions of the branches corresponding to a respective depth.

12. The apparatus of claim 11 , wherein:

the means for generating a branch overlap image channel representing a pixelwise probability that the branches overlap comprises means for generating the branch overlap image channel using a trained image to image network;

the means for generating a set of branch orientation image channels each associated with one of a plurality of orientations comprises means for generating the set of branch orientation image channels using a trained fully convolutional neural network; and

the means for generating a multi-channel depth image based on the branch overlap image channel and the set of branch orientation image channels comprises means for generating the multi-channel depth image using another trained image to image network.

13. The apparatus of claim 11 , wherein the plurality of orientations comprises a vertical orientation, a horizontal orientation, a first diagonal orientation, and a second diagonal orientation.

14. The apparatus of claim 11 , wherein the means for generating a set of branch orientation image channels each associated with one of a plurality of orientations comprises:

means for extracting a plurality of patches from the medical image;

for each particular patch of the plurality of patches:

means for applying a trained fully convolutional neural network to the particular patch to determine a probability, for each respective orientation of the plurality of orientations, that a pixel associated with the particular patch depicts the branches in the respective orientation; and

for each respective orientation, means for combining the probability for each pixel associated with each particular patch for the respective orientation to generate the set of branch orientation image channels.

15. A non-transitory computer readable medium storing computer program instructions, the computer program instructions when executed by a processor cause the processor to perform operations comprising:

receiving a medical image comprising branches of one or more vessels;

generating a branch overlap image channel representing a pixelwise probability that the branches overlap;

generating a set of branch orientation image channels each associated with one of a plurality of orientations, each branch orientation image channel representing a pixelwise probability that the branches are oriented in its associated orientation; and

generating a multi-channel depth image based on the branch overlap image channel and the set of branch orientation image channels, each channel of the multi-channel depth image comprising portions of the branches corresponding to a respective depth.

16. The non-transitory computer readable medium of claim 15 , wherein:

generating a branch overlap image channel representing a pixelwise probability that the branches overlap comprises generating the branch overlap image channel using a first trained machine learning network;

generating a set of branch orientation image channels each associated with one of a plurality of orientations comprises generating the set of branch orientation image channels using a second trained machine learning network; and

generating a multi-channel depth image based on the branch overlap image channel and the set of branch orientation image channels comprises generating the multi-channel depth image using a third trained machine learning network.

17. The non-transitory computer readable medium of claim 15 , the operations further comprising:

highlighting pixels in the branch overlap image channel based on the pixelwise probability that the branches overlap based on the medical image; and

highlighting pixels in the set of branch orientation image channels based on the pixelwise probability that the branches are oriented in their associated orientation.

18. The non-transitory computer readable medium of claim 15 , wherein generating a multi-channel depth image based on the branch overlap image channel and the set of branch orientation image channels comprises:

generating the multi-channel depth image based on coded high level features previously generated from another branch overlap image channel and another set of branch orientation image channels, the other branch overlap image channel and the other set of branch orientation image channels being generated from another medical image.

19. The non-transitory computer readable medium of claim 18 , wherein generating the multi-channel depth image based on coded high level features previously generated from another branch overlap image channel and another set of branch orientation image channels comprises:

generating the multi-channel depth image using a long short-term memory network.

20. The non-transitory computer readable medium of claim 15 , wherein generating a multi-channel depth image based on the branch overlap image channel and the set of branch orientation image channels comprises:

generating the multi-channel depth image based on pixel intensity values of the medical image.