Dense body marker estimation from camera data for patient positioning in medical imaging
Machine learning is used to train a network to estimate a three-dimensional (3D) body surface and body regions of a patient from surface images of the patient. The estimated 3D body surface of the patient is used to determine an isocenter of the patient. The estimated body regions are used to generate heatmaps representing visible body region boundaries and unseen body region boundaries of the patient. The estimation of 3D body surfaces, the determined patient isocenter, and the estimated body region boundaries may assist in planning a medical scan, including automatic patient positioning.
1. A method for estimating a three-dimensional body surface of a patient located on a moveable patient table in a medical imaging system, the method comprising:
capturing, with a sensor, a sequence of surface images of the patient;
estimating, by a processor, the three-dimensional body surface of the patient by a machine-learned network in response to input of the surface images to the machine-learned network;
automatically determining, by the processor, an isocenter of the patient based on a regression of the estimated three-dimensional body surface of the patient and based on a position of the moveable patient table; and
positioning, by the processor, the patient via the moveable patient table based on the isocenter of the patient.
2. The method of claim 1 wherein capturing comprises capturing with the sensor being a camera.
3. The method of claim 2 wherein capturing with the camera comprises capturing the surface images as red-green-blue (RGB) images.
4. The method of claim 1 wherein estimating further comprises:
estimating dense body markers for a first surface image of the sequence of surface images, the first surface image corresponding to a first imaged body portion of the patient;
reconstructing a first three-dimensional partial body surface for the first imaged body portion of the patient based on the first surface image;
estimating dense body markers for a second surface image of the sequence of surface images, the second surface image corresponding to a second imaged body portion of the patient;
reconstructing a second three-dimensional partial body surface for the second imaged body portion of the patient based on the second surface image; and
reconstructing the three-dimensional body surface of the patient from at least the first three-dimensional partial body surface and the second three-dimensional partial body surface using a recurrent model.
5. The method of claim 4 wherein the first imaged body portion of the patient and the second imaged body portion of the patient overlap.
6. The method of claim 4 wherein reconstructing the first three-dimensional partial body surface comprises:
partitioning the first surface image into a plurality of parts, wherein the plurality of parts corresponds to different body parts of the first imaged body portion of the patient, and wherein each part of the plurality of parts of the first surface image comprises a plurality of pixels;
assigning a classification to each pixel of the plurality of pixels of the first surface image; and
determining three-dimensional surface coordinates for each pixel of the plurality of pixels.
7. The method of claim 6 wherein assigning comprises assigning each pixel to a background class or a body part class, wherein the body part class corresponds to the different body parts of the first imaged body portion of the patient.
8. The method of claim 6 wherein partitioning and assigning comprises partitioning and assigning with at least one fully convolutional network with an encoder and a decoder, and wherein the determining comprises determining with at least one regressor.
9. The method of claim 8 wherein the at least one regressor comprises a plurality of regressors, wherein each regressor of the plurality of regressors corresponds to one of the parts of the plurality of parts of the first surface image, and wherein each regressor of the plurality of regressors is separately trained.
10. The method of claim 4 wherein the recurrent model comprises a recurrent neural network comprising Long Short-Term Memory (LSTM) units.
11. The method of claim 4 wherein the recurrent model was trained end-to-end.
12. A medical imaging system for estimating a three-dimensional body surface of a patient, the system comprising:
a sensor configured to capture a sequence of surface images of the patient;
a processor configured to:
apply a machine-learned neural network to the sequence of surface images, the machine-learned neural network having been trained to estimate the three-dimensional body surface of the patient from the sequence of surface images; and
automatically determine an isocenter of the patient based on a regression of the estimated three-dimensional body surface of the patient; and
a moveable patient table configured to automatically position the patient based on the isocenter of the patient and based on a position of the moveable patient table.
13. The medical imaging system of claim 12 wherein the machine-learned network was trained to:
estimate dense body markers for a first surface image of the sequence of surface images, the first surface image corresponding to a first imaged body portion of the patient;
reconstruct a first three-dimensional partial body surface for the first imaged body portion of the patient based on the first surface image;
estimate dense body markers for a second surface image of the sequence of surface images, the second surface image corresponding to a second imaged body portion of the patient;
reconstruct a second three-dimensional partial body surface for the second imaged body portion of the patient based on the second surface image; and
reconstruct the three-dimensional body surface of the patient from at least the first three-dimensional partial body surface and the second three-dimensional partial body surface using a recurrent model.
14. The medical imaging system of claim 13 wherein the recurrent model was trained end-to-end and comprises a recurrent neural network comprising Long Short-Term Memory (LSTM) units.
15. The medical imaging system of claim 13 wherein to reconstruct the first three-dimensional partial body surface, the machine-learned network was trained to:
partition the first surface image into a plurality of parts, wherein the plurality of parts corresponds to different body parts of the first imaged body portion of the patient, and wherein each part of the plurality of parts of the first surface image comprises a plurality of pixels;
assign a classification to each pixel of the plurality of pixels of the first surface image; and
determine three-dimensional surface coordinates for each pixel of the plurality of pixels.
16. The medical imaging system of claim 15 wherein the machine-learned network comprises a fully convolutional network with an encoder and a decoder, wherein the machine-learned network uses a plurality of separately trained regressors, wherein each regressor of the plurality of regressors corresponds to one of the parts of the plurality of parts of the first surface image, and wherein each regressor determines the three-dimensional surface coordinates for each pixel of the plurality of pixels associated with the corresponding part of the first surface image.