Unsupervised deep representation learning for fine-grained body part recognition

1. A method for deep learning based fine-grained body part recognition in medical imaging data, comprising:

training a paired convolutional neural network (P-CNN) for slice ordering based on unlabeled training medical image volumes; wherein training the P-CNN for slice ordering based on the unlabeled training medical image volumes comprises:

randomly sampling transversal slice pairs from the unlabeled training medical image volumes, wherein each transversal slice pair is randomly sampled from the same training volume, and

training the P-CNN to predict a relative order of a pair of transversal slices of a medical imaging volume based on the randomly sampled transversal slice pairs, wherein the P-CNN includes two identical sub-networks for a first plurality of layers, each to extract feature from a respective slice of the pair of transversal slices, and global final layers to fuse outputs of the sub-networks and calculate a binary classification result regarding the relative order of the pair of transversal slices; and

training a convolutional neural network (CNN) for fine-grained body part recognition by fine-tuning learned weights of the trained P-CNN for slice ordering.

2. The method of claim 1 , wherein the CNN for fine-grained body part recognition includes a first plurality of layers for extracting features from an input transversal slice of a medical imaging volume and a set of final regression layers for calculating a normalized height score for the input transversal slice corresponding to a normalized height of the input transversal slice in a human body.

3. The method of claim 2 , wherein training a convolutional neural network (CNN) for fine-grained body part recognition by fine-tuning learned weights of the trained P-CNN for slice ordering comprises:

initializing weights of the first plurality of layers of the CNN for fine-grained body part recognition using learned weights of the first plurality of layers of one of the sub-networks in the trained P-CNN for slice ordering;

randomly initializing weights of the final regression layers of the CNN; and

fine-tuning the weights of all of the layers of the CNN based on annotated training medical image volumes with a reduced learning rate of the first plurality of layers.

4. A method for deep learning based fine-grained body part recognition in medical imaging data, comprising:

training a paired convolutional neural network (P-CNN) for slice ordering based on unlabeled training medical image volumes; and

training a convolutional neural network (CNN) for fine-grained body part recognition by fine-tuning learned weights of the trained P-CNN for slice ordering, wherein training the CNN for fine-grained body part recognition by fine-tuning the learned weights of the trained P-CNN for slice ordering comprises:

generating a normalized body height model based on locations of a set of anatomical landmarks in at least a subset of annotated medical imaging volumes,

calculating normalized height scores for transversal slices of the annotated medical imaging volumes based on the normalized body height model, and

training the CNN for fine-grained body part recognition by fine-tuning the learned weights of the trained P-CNN for slice ordering based on the transversal slices of the annotated medical imaging volumes and the normalized height scores calculated for the transversal slices.

5. The method of claim 4 , wherein the normalized body height model normalizes a height of a human body between values of 0 and 1 and includes average normalized height values for each of the set of anatomical landmarks in at least the subset of the annotated medical imaging volumes.

6. The method of claim 5 , wherein calculating normalized height scores for transversal slices of the annotated medical imaging volumes based on the normalized body height model comprises:

for each of the annotated medical imaging volumes:

assigning a respective normalized height value for the location of each of the set of anatomical landmarks as the average normalized height value of that anatomical landmark in the normalized body height model; and

interpolating between the anatomical landmarks in the set of anatomical landmarks to determine normalized height values for the transversal slices between the anatomical landmark locations.

7. The method of claim 6 , wherein the set of anatomical landmarks comprises top of a head, neck, lung, spine, knee, and foot landmarks.

8. An apparatus for deep learning based fine-grained body part recognition in medical imaging data, comprising:

means for training a paired convolutional neural network (P-CNN) for slice ordering based on unlabeled training medical image volumes, wherein the means for training the P-CNN for slice ordering based on the unlabeled training medical image volumes comprises:

means for randomly sampling transversal slice pairs from the unlabeled training medical image volumes, wherein each transversal slice pair is randomly sampled from the same training volume, and

means for training the P-CNN to predict a relative order of a pair of transversal slices of a medical imaging volume based on the randomly sampled transversal slice pairs, wherein the P-CNN includes two identical sub-networks for a first plurality of layers, each to extract feature from a respective slice of the pair of transversal slices, and global final layers to fuse outputs of the sub-networks and calculate a binary classification result regarding the relative order of the pair of transversal slices; and

means for training a convolutional neural network (CNN) for fine-grained body part recognition by fine-tuning learned weights of the trained P-CNN for slice ordering.

9. The apparatus of claim 8 , wherein the CNN for fine-grained body part recognition includes a first plurality of layers for extracting features from an input transversal slice of a medical imaging volume and a set of final regression layers for calculating a normalized height score for the input transversal slice corresponding to a normalized height of the input transversal slice in a human body.

10. The apparatus of claim 9 , wherein the means for training a convolutional neural network (CNN) for fine-grained body part recognition by fine-tuning learned weights of the trained P-CNN for slice ordering comprises:

means for initializing weights of the first plurality of layers of the CNN for fine-grained body part recognition using learned weights of the first plurality of layers of one of the sub-networks in the trained P-CNN for slice ordering;

means for randomly initializing weights of the final regression layers of the CNN; and

means for fine-tuning the weights of all of the layers of the CNN based on annotated training medical image volumes with a reduced learning rate of the first plurality of layers.

11. An apparatus for deep learning based fine-grained body part recognition in medical imaging data, comprising:

means for training a paired convolutional neural network (P-CNN) for slice ordering based on unlabeled training medical image volumes; and

means for training a convolutional neural network (CNN) for fine-grained body part recognition by fine-tuning learned weights of the trained P-CNN for slice ordering, wherein the means for training the CNN for fine-grained body part recognition by fine-tuning the learned weights of the trained P-CNN for slice ordering comprises:

means for generating a normalized body height model based on locations of a set of anatomical landmarks in at least a subset of annotated medical imaging volumes,

means for calculating normalized height scores for transversal slices of the annotated medical imaging volumes based on the normalized body height model, and

means for training the CNN for fine-grained body part recognition by fine-tuning the learned weights of the trained P-CNN for slice ordering based on the transversal slices of the annotated medical imaging volumes and the normalized height scores calculated for the transversal slices.

12. The apparatus of claim 11 , wherein the normalized body height model normalizes a height of a human body between values of 0 and 1 and includes average normalized height values for each of the set of anatomical landmarks in at least the subset of the annotated medical imaging volumes.

13. The apparatus of claim 12 , wherein the means for calculating normalized height scores for transversal slices of the annotated medical imaging volumes based on the normalized body height model comprises:

means for assigning a respective normalized height value for the location of each of the set of anatomical landmarks in each of the annotated medical imaging volumes as the average normalized height value of that anatomical landmark in the normalized body height model; and

means for interpolating between the anatomical landmarks in the set of anatomical landmarks to determine normalized height values for the transversal slices between the anatomical landmark locations in each of the annotated medical imaging volumes.

14. A non-transitory computer readable medium storing computer program instructions for deep learning based fine-grained body part recognition in medical imaging data, the computer program instructions when executed on a processor cause the processor to perform operations comprising:

training a paired convolutional neural network (P-CNN) for slice ordering based on unlabeled training medical image volumes, wherein training the P-CNN for slice ordering based on the unlabeled training medical image volumes comprises:

randomly sampling transversal slice pairs from the unlabeled training medical image volumes, wherein each transversal slice pair is randomly sampled from the same training volume, and

training the P-CNN to predict a relative order of a pair of transversal slices of a medical imaging volume based on the randomly sampled transversal slice pairs, wherein the P-CNN includes two identical sub-networks for a first plurality of layers, each to extract feature from a respective slice of the pair of transversal slices, and global final layers to fuse outputs of the sub-networks and calculate a binary classification result regarding the relative order of the pair of transversal slices; and

training a convolutional neural network (CNN) for fine-grained body part recognition by fine-tuning learned weights of the trained P-CNN for slice ordering.

15. The non-transitory computer readable medium of claim 14 , wherein the CNN for fine-grained body part recognition includes a first plurality of layers for extracting features from an input transversal slice of a medical imaging volume and a set of final regression layers for calculating a normalized height score for the input transversal slice corresponding to a normalized height of the input transversal slice in a human body.

16. The non-transitory computer readable medium of claim 15 , wherein training a convolutional neural network (CNN) for fine-grained body part recognition by fine-tuning learned weights of the trained P-CNN for slice ordering comprises:

initializing weights of the first plurality of layers of the CNN for fine-grained body part recognition using learned weights of the first plurality of layers of one of the sub-networks in the trained P-CNN for slice ordering;

randomly initializing weights of the final regression layers of the CNN; and

fine-tuning the weights of all of the layers of the CNN based on annotated training medical image volumes with a reduced learning rate of the first plurality of layers.

17. A non-transitory computer readable medium storing computer program instructions for deep learning based fine-grained body part recognition in medical imaging data, the computer program instructions when executed on a processor cause the processor to perform operations comprising:

training a paired convolutional neural network (P-CNN) for slice ordering based on unlabeled training medical image volumes; and

training a convolutional neural network (CNN) for fine-grained body part recognition by fine-tuning learned weights of the trained P-CNN for slice ordering, wherein training the CNN for fine-grained body part recognition by fine-tuning the learned weights of the trained P-CNN for slice ordering comprises:

generating a normalized body height model based on locations of a set of anatomical landmarks in at least a subset of annotated medical imaging volumes,

calculating normalized height scores for transversal slices of the annotated medical imaging volumes based on the normalized body height model, and

training the CNN for fine-grained body part recognition by fine-tuning the learned weights of the trained P-CNN for slice ordering based on the transversal slices of the annotated medical imaging volumes and the normalized height scores calculated for the transversal slices.

18. The non-transitory computer readable medium of claim 17 , wherein the normalized body height model normalizes a height of a human body between values of 0 and 1 and includes average normalized height values for each of the set of anatomical landmarks in at least the subset of the annotated medical imaging volumes.

19. The non-transitory computer readable medium of claim 18 , wherein calculating normalized height scores for transversal slices of the annotated medical imaging volumes based on the normalized body height model comprises:

for each of the annotated medical imaging volumes:

assigning a respective normalized height value for the location of each of the set of anatomical landmarks as the average normalized height value of that anatomical landmark in the normalized body height model; and

interpolating between the anatomical landmarks in the set of anatomical landmarks to determine normalized height values for the transversal slices between the anatomical landmark locations.