Synthetic parameterized computed tomography from surface data in medical imaging
Synthetic CT is estimated for planning or other purposes from surface data (e.g., depth camera information). The estimation uses parameterization, such as landmark and/or segmentation information, in addition to the surface data. In training and/or application, the parameterization may be used to correct the predicted CT volume. The CT volume may be predicted as a sub-part of the patient, such as estimating the CT volume for scanning one system, organ, or type of tissue separately from other system, organ, or type of tissue.
1. A method for computed tomography (CT) prediction from surface data in a medical imaging system, the method comprising:
capturing, with a sensor, an outer surface of a patient;
determining a segmentation and/or a landmark location;
generating, by an image processor, a first three-dimensional (3D) CT representation of the patient by a first machine-learned generative network in response to input of the surface data and the segmentation and/or landmark location to the first machine-learned generative network, the surface data being from an output of the sensor for the outer surface, the first machine-learned generative network comprising a generator architecture with an encoder and following decoder, where a beginning of the encoder has inputs for both (1) the segmentation and/or landmark location and for (2) the surface data and the decoder has an output for the first 3D CT representation, the segmentation different than the surface data; and
displaying, by a display device, an image from the first 3D CT representation.
2. The method of claim 1 wherein capturing comprises capturing with the sensor being a depth sensor.
3. The method of claim 1 wherein capturing comprises capturing with the sensor being a camera where the surface data based on optical measurements.
4. The method of claim 1 wherein determining comprises determining the segmentation, and wherein generating comprises generating in response to the input of the surface data and the segmentation.
5. The method of claim 1 wherein determining comprises determining the segmentation and/or landmark location from scan data from a different medical imaging modality than CT.
6. The method of claim 1 wherein determining comprises determining the segmentation and the landmark location, and wherein generating comprises generating in response to the input of the surface data, the segmentation, and the landmark location.
7. The method of claim 1 wherein determining comprises determining with a second machine-learned generative network, the second machine-learned generative network outputting a segmentation map and/or landmark location map in response to input of the surface data and a second 3D CT representation of the patient, and wherein generating comprises generating by the first machine-learned generative network in response to input of the segmentation map as the segmentation and/or the landmark location map as the landmark location and input of the surface data.
8. The method of claim 7 further comprising forming the second 3D CT representation from an output of a third machine-learned generative network.
9. The method of claim 7 further comprising forming the second 3D CT representation from an output of the first machine-learned generative network.
10. The method of claim 9 wherein generating comprises iteratively using the first and second machine-learned generative networks.
11. The method of claim 1 wherein generating comprises generating the first 3D CT representation as a representation of first internal anatomy without second internal anatomy.
12. The method of claim 11 wherein generating further comprises generating a second 3D CT representation of the second internal anatomy without the first internal anatomy.
13. The method of claim 11 wherein generating comprises generating the first 3D CT representation as a voxel or mesh representation.
14. The method of claim 1 further comprising:
configuring a medical scanner based on the first 3D CT representation; and
imaging, by the medical scanner, the patient as configured based on the first 3D CT representation.
15. A method for computed tomography (CT) prediction from surface data in a medical imaging system, the method comprising:
capturing, with a sensor, an outer surface of a patient;
generating, by an image processor, a first three-dimensional (3D) CT representation by first and second machine-learned networks in response to input of the surface data to beginnings of both the first and second machine-learned networks, the surface data being from an output of the sensor for the outer surface, the first machine-learned network outputting a spatial segmentation based on the surface data, and the second machine-learned network outputting the first 3D CT representation based on the surface data and the spatial segmentation, the first and second machine-learned networks having been previously trained, the second machine-learned network comprising a generator having inputs for both the surface data and the spatial segmentation at the beginning of the generator; and
displaying, by a display device, an image from the first 3D CT representation.
16. The method of claim 15 wherein the first machine-learned network is configured to output the spatial segmentation and a landmark map, and the second machine-learned network is configured to output based on the surface data, the spatial segmentation, and the landmark map.
17. The method of claim 15 wherein generating comprises generating the first 3D CT representation from one of one or more output channels, each output channel representing different ones of only muscle, only skeleton, only vessel, only organ, and only a tissue type.