Method and system for model-based fusion of computed tomography and non-contrasted C-arm computed tomography
View Patent ↗A method and system for model-based fusion of multi-modal volumetric images is disclosed. A first patient-specific model of an anchor anatomical structure is detected in a first medical image acquired using a first imaging modality, and a second patient-specific model of the anchor anatomical structure is detected in a second medical image acquired using a second imaging modality. A weighted mapping function is determined based on the first patient-specific model of the anchor anatomical structure and the second patient-specific model of the anchor anatomical structure using learned weights to minimize mapping error with respect to a target anatomical structure. The target anatomical structure from the first medical image to the second medical image using the weighted mapping function. In an application of this model-based fusion to transcatheter valve therapies, the trachea bifurcation is used as the anchor anatomical structure and the aortic valve is the target anatomical structure.
1. A method for fusing a first medical image acquired using a first imaging modality and a second medical image acquired using a second imaging modality, comprising:
detecting a first patient-specific model of an anchor anatomical structure in the first medical image and a second patient-specific model of the anchor anatomical structure in the second medical image;
determining a weighted mapping function based on the first patient-specific model of the anchor anatomical structure and the second patient-specific model of the anchor anatomical structure using learned weights to minimize mapping error with respect to a target anatomical structure other than the anchor anatomical structure; and
mapping the target anatomical structure from the first medical image to the second medical image using the weighted mapping function.
2. The method of claim 1 , wherein detecting a first patient-specific model of an anchor anatomical structure in the first medical image and a second patient-specific model of the anchor anatomical structure in the second medical image comprises:
detecting each of the first and second models of the anchor anatomical structure using marginal space learning.
3. The method of claim 1 , wherein the anchor anatomical structure is visible the first and second imaging medical images without a contrast agent and the target anatomical structure is visible is the first medical image without a contrast agent not visible in the second medical image without a contrast agent.
4. The method of claim 1 , wherein determining a weighted mapping function based on the first patient-specific model of the anchor anatomical structure and the second patient-specific model of the anchor anatomical structure using learned weights to minimize mapping error with respect to a target anatomical structure comprises:
estimating a rigid transformation between the first patient-specific model of the anchor anatomical structure and the second patient-specific model of the anchor anatomical structure using a weighted least squares solution in which error values between transformed mesh points of the second patient-specific model of the anchor anatomy and corresponding mesh points of the first patient-specific model of the anchor anatomy are weighted using learned weights associated with the mesh points of the model of the anchor anatomical structure.
5. The method of claim 4 , wherein the learned weights associated with the mesh points of the model of the anchor anatomical structure are learned based on ground truth mappings of the target anatomical structure in a set of training data.
6. The method of claim 1 , wherein mapping the target anatomical structure from the first medical image to the second medical image using the weighted mapping function comprises:
mapping the first medical image to the second medical image using the weighted mapping function.
7. The method of claim 1 , wherein mapping the target anatomical structure from the first medical image to the second medical image using the weighted mapping function comprises:
mapping a region of the first medical image corresponding to the target anatomical structure to the second medical image using the weighted mapping function.
8. The method of claim 1 , wherein mapping the target anatomical structure from the first medical image to the second medical image using the weighted mapping function comprises:
mapping a patient-specific model of the target anatomical structure detected in the first medical image to the second medical image.
9. The method of claim 1 , wherein the anchor anatomical structure is a trachea bifurcation and the target anatomical structure is an aortic valve.
10. The method of claim 9 , wherein detecting a first patient-specific model of an anchor anatomical structure in the first medical image and a second patient-specific model of the anchor anatomical structure in the second medical image comprises:
for each of the first and second images, detecting a patient-specific trachea bifurcation model in each of the first and second medical images by:
detecting a global location of the trachea bifurcation model in the image,
detecting a plurality of trachea bifurcation landmarks in the image based on the detected global location of the trachea bifurcation model; and
fitting a surface model of the trachea bifurcation to the image.
11. The method of claim 10 , wherein detecting a plurality of trachea bifurcation landmarks in the image based on the detected global location of the trachea bifurcation model comprises:
detecting a trachea airway bifurcation point, a trachea lower-left airway branching point, a trachea lower-right airway branching point, and a trachea upper center airway point, using trained classifiers constrained by the detected global location of the trachea bifurcation model.
12. The method of claim 10 , wherein the surface model of the trachea bifurcation includes a trachea upper airway mesh, a trachea lower-left airway mesh, a trachea lower-right airway mesh, a trachea bifurcation face mesh, and a trachea bifurcation back mesh, and fitting a surface model of the trachea bifurcation to the image comprises:
generating a plurality of trachea bifurcation surface model shapes from a learned statistical shape model;
projecting each of the plurality of trachea bifurcation surface model shapes to an image space of image using the detected global location of the trachea bifurcation model, resulting in a plurality of surface model hypotheses; and
selecting one of the surface model hypotheses as the surface model of the trachea bifurcation using a trained classifier.
13. The method of claim 9 , wherein the learned weights are a set of weights associated with mesh points of a trachea bifurcation model that are learned based on ground truth mappings of aortic valve hinges and aortic valve commissures in a set of training data.
14. The method of claim 9 , wherein mapping the target anatomical structure from the first medical image to the second medical image using the weighted mapping function comprises:
mapping a patient-specific aortic valve model detected in the first medical image to the second medical image using the weighted mapping function.
15. The method of claim 1 , wherein the first medical image is a pre-operative computed tomography (CT) image and the second medical image is an intro-operative 3D C-arm CT image.
16. An apparatus for fusing a first medical image acquired using a first imaging modality and a second medical image acquired using a second imaging modality, comprising:
means for detecting a first patient-specific model of an anchor anatomical structure in the first medical image and a second patient-specific model of the anchor anatomical structure in the second medical image;
means for determining a weighted mapping function based on the first patient-specific model of the anchor anatomical structure and the second patient-specific model of the anchor anatomical structure using learned weights to minimize mapping error with respect to a target anatomical structure other than the anchor anatomical structure; and
means for mapping the target anatomical structure from the first medical image to the second medical image using the weighted mapping function.
17. The apparatus of claim 16 , wherein the means for determining a weighted mapping function based on the first patient-specific model of the anchor anatomical structure and the second patient-specific model of the anchor anatomical structure using learned weights to minimize mapping error with respect to a target anatomical structure comprises:
means for estimating a rigid transformation between the first patient-specific model of the anchor anatomical structure and the second patient-specific model of the anchor anatomical structure using a weighted least squares solution in which error values between transformed mesh points of the second patient-specific model of the anchor anatomy and corresponding mesh points of the first patient-specific model of the anchor anatomy are weighted using learned weights associated with the mesh points of the model of the anchor anatomical structure.
18. The apparatus of claim 17 , wherein the learned weights associated with the mesh points of the model of the anchor anatomical structure are learned based on ground truth mappings of the target anatomical structure in a set of training data.
19. The apparatus of claim 16 , wherein the anchor anatomical structure is a trachea bifurcation and the target anatomical structure is an aortic valve.
20. The apparatus of claim 19 , wherein the means for detecting a first patient-specific model of an anchor anatomical structure in the first medical image and a second patient-specific model of the anchor anatomical structure in the second medical image comprises:
means for detecting a global location of the trachea bifurcation model in an image;
means for detecting a plurality of trachea bifurcation landmarks in the image based on the detected global location of the trachea bifurcation model; and
means for fitting a surface model of the trachea bifurcation to the image.
21. The apparatus of claim 19 , wherein the learned weights are a set of weights associated with mesh points of a trachea bifurcation model that are learned based on ground truth mappings of aortic valve hinges and aortic valve commissures in a set of training data.
22. The apparatus of claim 16 , wherein the first medical image is a pre-operative computed tomography (CT) image and the second medical image is an intro-operative 3D C-arm CT image.
23. A non-transitory computer readable medium storing computer program instructions for fusing a first medical image acquired using a first imaging modality and a second medical image acquired using a second imaging modality, the computer program instructions, when executed by a processor cause the processor to perform operations comprising:
detecting a first patient-specific model of an anchor anatomical structure in the first medical image and a second patient-specific model of the anchor anatomical structure in the second medical image;
determining a weighted mapping function based on the first patient-specific model of the anchor anatomical structure and the second patient-specific model of the anchor anatomical structure using learned weights to minimize mapping error with respect to a target anatomical structure other than the anchor anatomical structure; and
mapping the target anatomical structure from the first medical image to the second medical image using the weighted mapping function.
24. The non-transitory computer readable medium of claim 23 , wherein determining a weighted mapping function based on the first patient-specific model of the anchor anatomical structure and the second patient-specific model of the anchor anatomical structure using learned weights to minimize mapping error with respect to a target anatomical structure comprises:
estimating a rigid transformation between the first patient-specific model of the anchor anatomical structure and the second patient-specific model of the anchor anatomical structure using a weighted least squares solution in which error values between transformed mesh points of the second patient-specific model of the anchor anatomy and corresponding mesh points of the first patient-specific model of the anchor anatomy are weighted using learned weights associated with the mesh points of the model of the anchor anatomical structure.
25. The non-transitory computer readable medium of claim 24 , wherein the learned weights associated with the mesh points of the model of the anchor anatomical structure are learned based on ground truth mappings of the target anatomical structure in a set of training data.
26. The non-transitory computer readable medium of claim 23 , wherein mapping the target anatomical structure from the first medical image to the second medical image using the weighted mapping function comprises:
mapping a patient-specific model of the target anatomical structure detected in the first medical image to the second medical image.
27. The non-transitory computer readable medium of claim 23 , wherein the anchor anatomical structure is a trachea bifurcation and the target anatomical structure is an aortic valve.
28. The non-transitory computer readable medium of claim 26 , wherein detecting a first patient-specific model of an anchor anatomical structure in the first medical image and a second patient-specific model of the anchor anatomical structure in the second medical image comprises:
for each of the first and second images, detecting a patient-specific trachea bifurcation model in each of the first and second medical images by:
detecting a global location of the trachea bifurcation model in the image,
detecting a plurality of trachea bifurcation landmarks in the image based on the detected global location of the trachea bifurcation model; and
fitting a surface model of the trachea bifurcation to the image.
29. The non-transitory computer readable medium of claim 26 , wherein the learned weights are a set of weights associated with mesh points of a trachea bifurcation model that are learned based on ground truth mappings of aortic valve hinges and aortic valve commissures in a set of training data.
30. The non-transitory computer readable medium of claim 26 , wherein mapping the target anatomical structure from the first medical image to the second medical image using the weighted mapping function comprises:
mapping a patient-specific aortic valve model detected in the first medical image to the second medical image using the weighted mapping function.
31. The non-transitory computer readable medium of claim 23 , wherein the first medical image is a pre-operative computed tomography (CT) image and the second medical image is an intro-operative 3D C-arm CT image.