Generating volume predictions of three-dimensional volumes using slice features

1. A system, comprising a processor to:

receive a three-dimensional (3D) volume;

partition the 3D volume into slices;

generate, via a shared two-dimensional (2D) neural network, slice features based on the slices, wherein the shared 2D neural network is trained via deep supervision using a 2D auxiliary task component comprising a slice predictor that predicts slice predictions based on slice features generated by the shared 2D neural network to generate an auxiliary loss used for the deep supervision; and

generate, via a 3D predictor, a volume prediction based on a 3D feature volume comprising merged slice features, wherein the 3D predictor is coupled to a merger that merges the slice features from the shared 2D neural network, wherein the volume prediction is used to calculate a 3D prediction loss for training the shared 2D neural network.

2. The system of claim 1 , wherein the shared 2D neural network and the 3D predictor are jointly trained using a 2D prediction loss comprising the auxiliary loss computed based on the slice predictions and the 3D prediction loss computed based on the volume prediction.

3. The system of claim 1 , wherein the shared 2D neural network and the 3D predictor are jointly trained based on a total loss comprising a weighted sum of a 2D prediction loss and a 3D prediction loss.

4. The system of claim 1 , wherein the 3D predictor comprises a three-dimensional (3D) feature extractor to generate a three-dimensional (3D) feature map based on the slice features, and a volume predictor to generate a volume prediction based on the 3D feature map.

5. The system of claim 1 , wherein the processor is to receive an unannotated 3D volume and modify the shared 2D neural network and the 3D neural network using semi-supervised training.

6. A computer-implemented method, comprising:

receiving, via a processor, an annotated three-dimensional (3D) volume;

partitioning, via the processor, the annotated 3D volume into slices;

generating, via a two-dimensional (2D) neural network, slice features based on the slices, wherein the shared 2D neural network is trained via deep supervision using a 2D auxiliary task component comprising a slice predictor that predicts slice predictions based on slice features generated by the shared 2D neural network to generate an auxiliary loss used for the deep supervision;

fusing, via the processor, the slice features to generate a three-dimensional (3D) feature volume;

calculating, via the processor, a three-dimensional (3D) prediction loss based on a predicted volume generated, via a three-dimensional (3D) neural network, based on the 3D feature volume; and

modifying, via the processor, the shared 2D neural network and the 3D neural network based on the 3D prediction loss to produce a trained shared 2D neural network and a trained 3D neural network.

7. The computer-implemented method of claim 6 , comprising:

receiving, via the processor, a second three-dimensional (3D) volume;

partitioning, via the processor, the second 3D volume into slices;

generating, via the two-dimensional (2D) neural network, slice features based on the slices;

fusing, via the processor, the slice features to generate a 3D feature volume;

generating, via a trained three-dimensional (3D) feature extractor, a three-dimensional (3D) feature volume based on the 3D feature volume; and

generating, via a volume predictor, a volume prediction based on the 3D feature volume.

8. The computer-implemented method of claim 6 , comprising:

calculating, via the processor, a two-dimensional (2D) prediction loss based on a slice prediction generated, via the two-dimensional (2D) neural network, based on the slice features;

generating, via the processor, a final loss based on the 3D prediction loss and the 2D prediction loss; and

modifying, via the processor, the shared 2D neural network and the 3D neural network based on the final loss to produce an end-to-end trained hybrid neural network.

9. The computer-implemented method of claim 8 , comprising receiving, via the processor, a three-dimensional (3D) volume and generating, via a volume predictor of the end-to-end trained hybrid neural network, a volume prediction.

10. The computer-implemented method of claim 8 , wherein generating the final loss comprises calculating a weighted sum of the 2D prediction loss and the 3D prediction loss.

11. The computer-implemented method of claim 8 , comprising receiving an unannotated 3D volume, and modifying the shared 2D neural network and the 3D neural network using semi-supervised training.

12. A computer program product for training neural networks, the computer program product comprising a computer-readable storage medium having program code embodied therewith, wherein the computer-readable storage medium is not a transitory signal per se, the program code executable by a processor to cause the processor to:

receive an annotated three-dimensional (3D) volume;

partition the annotated 3D volume into slices;

generate slice features based on the slices via a shared 2D neural network trained via deep supervision using a 2D auxiliary task component comprising a slice predictor that predicts slice predictions based on slice features generated by the shared 2D neural network to generate an auxiliary loss used for the deep supervision;

merge the slice features to generate a three-dimensional (3D) feature volume;

calculate a three-dimensional (3D) prediction loss based on a predicted volume generated, via a three-dimensional (3D) neural network, based on the 3D feature volume; and

modify the shared 2D neural network and the 3D neural network based on the 3D prediction loss to produce a trained shared 2D neural network and a trained 3D neural network.

13. The computer program product of claim 12 , further comprising program code executable by the processor to:

receive a second three-dimensional (3D) volume;

partition the second 3D volume into slices;

generate slice features based on the slices;

generate a three-dimensional (3D) feature volume based on the slice features;

generate a 3D feature map based on the 3D feature volume; and

generate a volume prediction based on the 3D feature map.

14. The computer program product of claim 12 , further comprising program code executable by the processor to:

calculate a two-dimensional (2D) prediction loss based on a slice prediction generated, via the two-dimensional (2D) neural network, based on the slice features;

generate a final loss based on the 3D prediction loss and the 2D prediction loss; and

modify the shared 2D neural network and the 3D neural network based on the final loss to produce an end-to-end trained hybrid neural network.

15. The computer program product of claim 14 , further comprising program code executable by the processor to receive, a three-dimensional (3D) volume and generate, via a volume predictor of the end-to-end trained hybrid neural network, a volume prediction.

16. The computer program product of claim 14 , further comprising program code executable by the processor to calculate a final loss based on a weighted sum of the 2D prediction loss and the 3D prediction loss.

17. The computer program product of claim 12 , further comprising program code executable by the processor to concatenate the slice features to generate the 3D feature volume.

18. The computer program product of claim 12 , wherein the annotated 3D volume comprises a 3D medical image containing an annotated object.