Deep image-to-image recurrent network with shape basis for automatic vertebra labeling in large-scale 3D CT volumes

1. A method for automated vertebra localization and identification in a 3D computed tomography (CT) volume of a patient, comprising:

predicting initial vertebra locations in a 3D CT volume of a patient for a plurality of vertebrae corresponding to a plurality of vertebra labels using a trained deep image-to-image network (DI2IN);

refining the initial vertebra locations for the plurality of vertebrae predicted using the DI2IN using a trained recurrent neural network, resulting in an updated set of vertebra locations for the plurality of vertebrae corresponding to the plurality of vertebrae labels; and

determining final vertebra locations in the 3D CT volume for the plurality of vertebrae corresponding to the plurality of vertebra labels by refining the updated set of vertebra locations using a trained shape-basis deep neural network.

2. The method of claim 1 , wherein predicting initial vertebra locations in a 3D CT volume of a patient for a plurality of vertebrae corresponding to a plurality of vertebra labels using a trained deep image-to-image network (DI2IN) comprises:

inputting the 3D CT volume to the trained DI2IN; and

generating, by the trained DI2IN from the 3D CT volume, a plurality of probability maps, each corresponding to a respective one of the plurality of vertebra labels.

3. The method of claim 2 , wherein the trained DI2IN comprises a convolutional encoder-decoder trained to perform multi-channel voxel wise regression from the 3D CT volume to generate a respective one of the plurality of probability maps on each of a plurality of channels.

4. The method of claim 3 , wherein the trained DI2IN is trained based on training images and ground truth probability maps generated using a Gaussian distribution surrounding ground truth locations for the plurality vertebrae corresponding to the plurality of vertebra labels in each training image.

5. The method of claim 3 , wherein the trained DI2IN is trained using a multi-level deep supervision network including a plurality of branches diverging from decoder layers of the convolutional encoder-decoder to minimize, for each of the plurality of channels, a total loss function that combines a voxel-wise loss between an output layer of each of the plurality of branches a ground truth probability map and a voxel-wise loss between a final probability map output by the decoder and the ground truth probability map over a plurality of training samples.

6. The method of claim 2 , wherein the trained recurrent neural network is convolutional long short-term memory (LSTM) and refining the initial vertebra locations for the plurality of vertebrae predicted using the DI2IN using the trained recurrent neural network comprises:

sequentially inputting the plurality of probability maps generated by the DI2IN to the convolutional LSTM in an order based on respective one of the plurality of vertebra labels corresponding to each of the plurality of probability maps; and

generating, by the convolutional LSTM, an updated probability map for each sequentially input probability map generated by the DI2IN.

7. The method of claim 6 , wherein sequentially inputting the plurality of probability maps generated by the DI2IN to the convolutional LSTM in an order based on respective one of the plurality of vertebra labels corresponding to each of the plurality of probability maps comprises:

sequentially inputting the plurality of probability maps generated by the DI2IN to the convolutional LSTM in a forward order from a top vertebra label to a bottom vertebra label; and

sequentially inputting the plurality of probability maps generated by the DI2IN to the convolutional LSTM in a backward order from the bottom vertebra label to the top vertebra label.

8. The method of claim 7 , wherein generating, by the convolutional LSTM, an updated probability map for each sequentially input probability map generated by the DI2IN comprises:

generating, by the convolutional LSTM, a first updated probability map for each of plurality of probability maps generated by the DI2IN in response to sequentially inputting the plurality of probability maps in the forward order;

generating, by the convolutional LSTM, a second updated probability map for each of plurality of probability maps generated by the DI2IN in response to sequentially inputting the plurality of probability maps in the backward order; and

combining the first and second updated probability maps generated for each of the plurality of probability maps.

9. The method of claim 6 , wherein generating, by the convolutional LSTM, an updated probability map for each sequentially input probability map generated by the DI2IN comprises:

for each of the sequentially input probability map, generating the update probability map by the convolution LSTM based on the input probability map and a cell state and hidden state computed for a previous input one of the plurality of probability maps.

10. The method of claim 6 , wherein determining final vertebra locations in the 3D CT volume for the plurality of vertebrae corresponding to the plurality of vertebra labels by refining the updated set of vertebra locations using a trained shape-basis deep neural network comprises:

determining an input coordinate vector corresponding to the updated set of vertebra locations from the updated probability maps generated by the convolutional LSTM;

inputting the input coordinate vector to the trained shape-basis deep neural network; and

generating, by the trained shape-basis deep neural network, a coefficient vector that defines a refined coordinate vector corresponding to the final vertebra locations in the 3D volume as a linear combination of a shape-based dictionary learned from a set of training samples.

11. The method of claim 10 , wherein determining an input coordinate vector corresponding to the update set of vertebra locations from the updated probability maps generated by the convolutional LSTM is performed by a fully connected layer between the convolutional LSTM and the trained shape-basis network, and the DI2IN, the convolutional LSTM, and the shape-basis network are trained as an end-to-end network.

12. The method of claim 1 , wherein determining final vertebra locations in the 3D CT volume for the plurality of vertebrae corresponding to the plurality of vertebra labels by refining the updated set of vertebra locations using a trained shape-basis deep neural network comprises:

inputting coordinate vector corresponding to the updated set of vertebra locations to the trained shape-basis network; and

generating, by the trained shape-basis deep neural network, a coefficient vector that defines a refined coordinate vector corresponding to the final vertebra locations in the 3D volume as a linear combination of a shape-based dictionary learned from a set of training samples.

13. An apparatus for automated vertebra localization and identification in a 3D computed tomography (CT) volume of a patient, comprising:

means for predicting initial vertebra locations in a 3D CT volume of a patient for a plurality of vertebrae corresponding to a plurality of vertebra labels using a trained deep image-to-image network (DI2IN);

means for refining the initial vertebra locations for the plurality of vertebrae predicted using the DI2IN using a trained recurrent neural network, resulting in an updated set of vertebra locations for the plurality of vertebrae corresponding to the plurality of vertebrae labels; and

means for determining final vertebra locations in the 3D CT volume for the plurality of vertebrae corresponding to the plurality of vertebra labels by refining the updated set of vertebra locations using a trained shape-basis deep neural network.

14. The apparatus of claim 13 , wherein the means for predicting initial vertebra locations in a 3D CT volume of a patient for a plurality of vertebrae corresponding to a plurality of vertebra labels using a trained deep image-to-image network (DI2IN) comprises:

means for generating, by the trained DI2IN from the 3D CT volume, a plurality of probability maps, each corresponding to a respective one of the plurality of vertebra labels.

15. The apparatus of claim 14 , wherein the trained recurrent neural network is convolutional long short-term memory (LSTM) and the means for refining the initial vertebra locations for the plurality of vertebrae predicted using the DI2IN using the trained recurrent neural network comprises:

means for sequentially inputting the plurality of probability maps generated by the DI2IN to the convolutional LSTM in an order based on respective one of the plurality of vertebra labels corresponding to each of the plurality of probability maps; and

means for generating, by the convolutional LSTM, an updated probability map for each sequentially input probability map generated by the DI2IN.

16. The apparatus of claim 13 , wherein the means for determining final vertebra locations in the 3D CT volume for the plurality of vertebrae corresponding to the plurality of vertebra labels by refining the updated set of vertebra locations using a trained shape-basis deep neural network comprises:

means for generating, by the trained shape-basis deep neural network based on an input coordinate vector corresponding to the updated set of vertebra locations, a coefficient vector that defines a refined coordinate vector corresponding to the final vertebra locations in the 3D volume as a linear combination of a shape-based dictionary learned from a set of training samples.

17. A non-transitory computer readable medium storing computer program instructions for automated vertebra localization and identification in a 3D computed tomography (CT) volume of a patient, the computer program instructions when executed by a processor cause the processor to perform operation comprising:

predicting initial vertebra locations in a 3D CT volume of a patient for a plurality of vertebrae corresponding to a plurality of vertebra labels using a trained deep image-to-image network (DI2IN);

refining the initial vertebra locations for the plurality of vertebrae predicted using the DI2IN using a trained recurrent neural network, resulting in an updated set of vertebra locations for the plurality of vertebrae corresponding to the plurality of vertebrae labels; and

determining final vertebra locations in the 3D CT volume for the plurality of vertebrae corresponding to the plurality of vertebra labels by refining the updated set of vertebra locations using a trained shape-basis deep neural network.

18. The non-transitory computer readable medium of claim 17 , wherein predicting initial vertebra locations in a 3D CT volume of a patient for a plurality of vertebrae corresponding to a plurality of vertebra labels using a trained deep image-to-image network (DI2IN) comprises:

inputting the 3D CT volume to the trained DI2IN; and

generating, by the trained DI2IN from the 3D CT volume, a plurality of probability maps, each corresponding to a respective one of the plurality of vertebra labels.

19. The non-transitory computer readable medium of claim 18 , wherein the trained DI2IN comprises a convolutional encoder-decoder trained to perform multi-channel voxel wise regression from the 3D CT volume to generate a respective one of the plurality of probability maps on each of a plurality of channels.

20. The non-transitory computer readable medium of claim 18 , wherein the trained recurrent neural network is convolutional long short-term memory (LSTM) and refining the initial vertebra locations for the plurality of vertebrae predicted using the DI2IN using the trained recurrent neural network comprises:

sequentially inputting the plurality of probability maps generated by the DI2IN to the convolutional LSTM in an order based on respective one of the plurality of vertebra labels corresponding to each of the plurality of probability maps; and

generating, by the convolutional LSTM, an updated probability map for each sequentially input probability map generated by the DI2IN.

21. The non-transitory computer readable medium of claim 20 , wherein sequentially inputting the plurality of probability maps generated by the DI2IN to the convolutional LSTM in an order based on respective one of the plurality of vertebra labels corresponding to each of the plurality of probability maps comprises:

sequentially inputting the plurality of probability maps generated by the DI2IN to the convolutional LSTM in a forward order from a top vertebra label to a bottom vertebra label, and

sequentially inputting the plurality of probability maps generated by the DI2IN to the convolutional LSTM in a backward order from the bottom vertebra label to the top vertebra label; and

wherein generating, by the convolutional LSTM, an updated probability map for each sequentially input probability map generated by the DI2IN comprises:

generating, by the convolutional LSTM, a first updated probability map for each of plurality of probability maps generated by the DI2IN in response to sequentially inputting the plurality of probability maps in the forward order,

generating, by the convolutional LSTM, a second updated probability map for each of plurality of probability maps generated by the DI2IN in response to sequentially inputting the plurality of probability maps in the backward order, and

combining the first and second updated probability maps generated for each of the plurality of probability maps.

22. The non-transitory computer readable medium of claim 20 , wherein determining final vertebra locations in the 3D CT volume for the plurality of vertebrae corresponding to the plurality of vertebra labels by refining the updated set of vertebra locations using a trained shape-basis deep neural network comprises:

determining an input coordinate vector corresponding to the updated set of vertebra locations from the updated probability maps generated by the convolutional LSTM;

inputting the input coordinate vector to the trained shape-basis deep neural network; and

generating, by the trained shape-basis deep neural network, a coefficient vector that defines a refined coordinate vector corresponding to the final vertebra locations in the 3D volume as a linear combination of a shape-based dictionary learned from a set of training samples.

23. The non-transitory computer readable medium of claim 22 , wherein determining an input coordinate vector corresponding to the update set of vertebra locations from the updated probability maps generated by the convolutional LSTM is performed by a fully connected layer between the convolutional LSTM and the trained shape-basis network, and the DI2IN, the convolutional LSTM, and the shape-basis network are trained as an end-to-end network.