3D segmentation using space carving and 2D convolutional neural networks

1. One or more computing systems for generating a 3D segmentation of a target volume from views collected during a scan of a target volume, the one or more computing systems comprise:

a computer-readable storage medium that stores computer-executable instructions for controlling the one or more computing system, the instructions include instructions of:

a feature map layer that inputs the views and outputs a 2D multi-channel feature vector for each pixel of the views, the 2D multi-channel feature vectors for a view represent a 2D multi-channel feature map of that view; and

a 2D-to-3D transform layer inputs the 2D multi-channel feature maps and outputs the 3D multi-label map that represents a 3D segmentation of the target volume; and

a processor for executing the computer-executable instructions stored in the computer-readable storage medium.

2. The one or more computing systems of claim 1 wherein the feature map layer includes, for each view, a 2D CNN that inputs that view and outputs a 2D multi-channel feature vector for each pixel of that view.

3. The one or more computing systems of claim 2 wherein the 2D CNNs and the 2D-to-3D transform layer are trained using views of training volumes, each training volume labeled with a 3D segmentation.

4. The one or more computing systems of claim 3 wherein the 2D CNNs include weights that are learned based on a loss function that factors in the weights of the 2D CNNs in parallel.

5. The one or more computing systems of claim 1 wherein the 2D-to-3D transform layer includes a space carver layer that inputs the 2D multi-channel feature maps and outputs a 3D channel volume for each channel, the 3D channel volume for a channel includes a feature for each voxel that is derived from feature values of the 2D multi-label feature maps for that channel.

6. The one or more computing systems of claim 5 wherein the 2D-to-3D transform layer further includes a combining layer that combines the 3D channel volumes for the channels to generate the 3D multi-label map for the target volume.

7. The one or more computing systems of claim 1 further including instructions to generate a 3D image of the target volume based on the 3D multi-label map of the target volume.

8. The one or more computing systems of claim 1 wherein the feature map layer includes a 2D CNN that inputs each of the views.

9. The one or more computing systems of claim 1 wherein the feature map layer includes, for each view, a 2D CNN that inputs that view and outputs a 2D multi-channel feature vector for each pixel of that view and wherein each 2D CNN includes a downsampling path and an upsampling path.

10. The one or more computing systems of claim 9 wherein the downsampling path includes pooling operators and the upsampling path includes upsampling operators.

11. The one or more computing systems of claim 1 wherein the scan is an X-ray scan.

12. A method performed by one or more computing systems for generating a 3D segmentation of a target volume from views collected during a scan of a target volume, the method comprising:

accessing views of an X-ray scan of a target volume;

generating a 2D multi-channel feature map for each view, the 2D multi-channel feature map for a view includes a feature vector for pixel of the view with a feature value for each channel;

generating a 3D channel volume for each channel based on the 2D multi-channel feature maps;

generating a 3D multi-label map that represents a 3D segmentation of the target volume based on the 3D channel volumes; and

outputting an indication of the 3D multi-label map.

13. The method of claim 12 wherein the generating of the 2D multi-channel feature map for a view applies a 2D CNN to the view, each view having a separate 2D CNN are trained in parallel.

14. The method of claim 12 wherein the generating of a 3D channel volume for a channel applies a space carver technique to the feature values of the 2D multi-channel feature maps for that channel.

15. The method of claim 12 wherein the generating of the 3D multi-label map combines the values of the 3D channel volumes and applies a softmax function to generate a probability distribution for each voxel of the 3D multi-label map.

16. The method of claim 12 further comprising generating a 3D image of the target volume is based on the 3D multi-label map of the target volume.

17. A method performed by one or more computing systems for training a machine learning system to generate a 3D segmentation of a target volume from views of the target volume, the method comprising:

accessing training data represent training views of training volumes, each training volume labeled with a 3D segmentation of that training volume; and

until a termination condition is satisfied,

for each view, applying a 2D convolutional neural network (CNN) to that view where each 2D CNN outputs a 2D multi-channel feature vector for each pixel of that view, each 2D CNN having weights, the 2D multi-channel feature vectors for a view forming a 2D multi-channel feature map for that view;

for each channel, applying a space carver to each of the feature values of the 2D multi-channel feature maps to generate a 3D volume for each channel, the space carver having weights for each channel;

applying a combining layer to the 3D volumes for the channels to generate a 3D multi-label map representing a 3D segmentation of the image, the combining layer having weights;

generating new values for one or more of the weights; and

applying a loss function to determine whether the termination condition is satisfied.

18. The method of claim 17 wherein the views of a target volume represent views of an X-ray scan.

19. The method of claim 18 wherein the views are generated from X-ray scans of target volumes.

20. One or more computing systems for generating a 2D segmentation of a slice of a target volume from views collected during a scan of a target volume, the one or more computing systems comprise:

a computer-readable storage medium that stores computer-executable instructions for controlling the one or more computing system, the instructions include instructions of:

a feature map layer that inputs the views and outputs a 1D multi-channel feature vector for each pixel of the views that corresponds to the slice, the 1D multi-channel feature vectors of a view represents a 1D multi-channel feature map for that view; and

a 1D-to-2D transform layer inputs the 1D multi-channel feature maps and outputs a 2D multi-label map that represents the 2D segmentation of the slice of the target volume; and

a processor for executing the computer-executable instructions stored in the computer-readable storage medium.