Dilated fully convolutional network for 2D/3D medical image registration
A method and system for 3D/3D medical image registration. A digitally reconstructed radiograph (DRR) is rendered from a 3D medical volume based on current transformation parameters. A trained multi-agent deep neural network (DNN) is applied to a plurality of regions of interest (ROIs) in the DRR and a 2D medical image. The trained multi-agent DNN applies a respective agent to each ROI to calculate a respective set of action-values from each ROI. A maximum action-value and a proposed action associated with the maximum action value are determined for each agent. A subset of agents is selected based on the maximum action-values determined for the agents. The proposed actions determined for the selected subset of agents are aggregated to determine an optimal adjustment to the transformation parameters and the transformation parameters are adjusted by the determined optimal adjustment. The 3D medical volume is registered to the 2D medical image using final transformation parameters resulting from a plurality of iterations.
1. A method for automated computer-based registration of a 3D medical volume to a 2D medical image, comprising:
rendering a 2D digitally reconstructed radiograph (DRR) from the 3D medical volume based on current transformation parameters;
determining, by an intelligent artificial agent, an action-value for each of a plurality of possible actions based on a region of interest (ROI) in the DRR and a ROI in the 2D medical image, the plurality of possible actions corresponding to predetermined adjustments of the current transformation parameters, the plurality of possible actions including positive and negative translations along x, y, and z axes by a predetermined amount and positive and negative rotations about the x, y, and z axes by a predetermined amount;
selecting an action from the plurality of possible actions based on the action-values;
adjusting the current transformation parameters by applying the selected action to provide adjusted transformation parameters;
repeating the rendering, the determining, the selecting, and the adjusting steps for a plurality of iterations using the adjusted transformation parameters as the current transformation parameters; and
registering the 3D medical volume to the 2D medical image using the adjusted transformation parameters resulting from the plurality of iterations.
2. The method of claim 1 , wherein determining, by an intelligent artificial agent, an action-value for each of a plurality of possible actions based on a region of interest (ROI) in the DRR and a corresponding ROI in the 2D medical image comprises:
encoding a region of interest (ROI) in the DRR using a first convolutional neural network (CNN) to generate a first feature vector and separately encoding a corresponding ROI in the 2D medical image using a second CNN to generate a second feature vector;
concatenating the first feature vector and the second feature vector into a concatenated feature vector; and
decoding the concatenated feature vector to determine the action-value for each of the plurality of possible actions.
3. The method of claim 2 , wherein the first CNN and the second CNN have the same structure.
4. The method of claim 1 , wherein selecting an action from the plurality of possible actions based on the action-values comprises:
selecting the action from the plurality of possible actions having a highest action-value.
5. The method of claim 1 , wherein the ROI in the DRR and the ROI in the 2D medical image are centered at a location of the intelligent artificial agent.
6. The method of claim 1 , wherein the ROI in the DRR corresponds to the ROI in the 2D medical image.
7. The method of claim 1 , wherein the 3D medical volume is a CT (computed tomography) volume and the 2D medical image is an x-ray image.
8. An apparatus for automated computer-based registration of a 3D medical volume to a 2D medical image, comprising:
a processor; and
a memory storing computer program instructions, which when executed by the processor cause the processor to perform operations comprising:
rendering a 2D digitally reconstructed radiograph (DRR) from the 3D medical volume based on current transformation parameters;
determining, by an intelligent artificial agent, an action-value for each of a plurality of possible actions based on a region of interest (ROI) in the DRR and a ROI in the 2D medical image, the plurality of possible actions corresponding to predetermined adjustments of the current transformation parameters, the plurality of possible actions including positive and negative translations along x, y, and z axes by a predetermined amount and positive and negative rotations about the x, y, and z axes by a predetermined amount;
selecting an action from the plurality of possible actions based on the action-values;
adjusting the current transformation parameters by applying the selected action to provide adjusted transformation parameters;
repeating the rendering, the determining, the selecting, and the adjusting steps for a plurality of iterations using the adjusted transformation parameters as the current transformation parameters; and
registering the 3D medical volume to the 2D medical image using the adjusted transformation parameters resulting from the plurality of iterations.
9. The apparatus of claim 8 , wherein determining, by an intelligent artificial agent, an action-value for each of a plurality of possible actions based on a region of interest (ROI) in the DRR and a corresponding ROI in the 2D medical image comprises:
encoding a region of interest (ROI) in the DRR using a first convolutional neural network (CNN) to generate a first feature vector and separately encoding a corresponding ROI in the 2D medical image using a second CNN to generate a second feature vector;
concatenating the first feature vector and the second feature vector into a concatenated feature vector; and
decoding the concatenated feature vector to determine the action-value for each of the plurality of possible actions.
10. The apparatus of claim 9 , wherein the first CNN and the second CNN have the same structure.
11. The apparatus of claim 8 , wherein selecting an action from the plurality of possible actions based on the action-values comprises:
selecting the action from the plurality of possible actions having a highest action-value.
12. The apparatus of claim 8 , wherein the ROI in the DRR and the ROI in the 2D medical image are centered at a location of the intelligent artificial agent.
13. A non-transitory computer readable medium storing computer program instructions for automated computer-based registration of a 3D medical volume to a 2D medical image, wherein the computer program instructions when executed by a processor cause the processor to perform operations comprising:
rendering a 2D digitally reconstructed radiograph (DRR) from the 3D medical volume based on current transformation parameters;
determining, by an intelligent artificial agent, an action-value for each of a plurality of possible actions based on a region of interest (ROI) in the DRR and a ROI in the 2D medical image, the plurality of possible actions corresponding to predetermined adjustments of the current transformation parameters, the plurality of possible actions including positive and negative translations along x, y, and z axes by a predetermined amount and positive and negative rotations about the x, y, and z axes by a predetermined amount;
selecting an action from the plurality of possible actions based on the action-values;
adjusting the current transformation parameters by applying the selected action to provide adjusted transformation parameters;
repeating the rendering, the determining, the selecting, and the adjusting steps for a plurality of iterations using the adjusted transformation parameters as the current transformation parameters; and
registering the 3D medical volume to the 2D medical image using the adjusted transformation parameters resulting from the plurality of iterations.
14. The non-transitory computer readable medium of claim 13 , wherein determining, by an intelligent artificial agent, an action-value for each of a plurality of possible actions based on a region of interest (ROI) in the DRR and a corresponding ROI in the 2D medical image comprises:
encoding a region of interest (ROI) in the DRR using a first convolutional neural network (CNN) to generate a first feature vector and separately encoding a corresponding ROI in the 2D medical image using a second CNN to generate a second feature vector;
concatenating the first feature vector and the second feature vector into a concatenated feature vector; and
decoding the concatenated feature vector to determine the action-value for each of the plurality of possible actions.
15. The non-transitory computer readable medium of claim 14 , wherein the first CNN and the second CNN have the same structure.
16. The non-transitory computer readable medium of claim 13 , wherein the ROI in the DRR corresponds to the ROI in the 2D medical image.
17. The non-transitory computer readable medium of claim 13 , wherein the 3D medical volume is a CT (computed tomography) volume and the 2D medical image is an x-ray image.