Registering intra-operative images transformed from pre-operative images of different imaging-modality for computer assisted navigation during surgery
A computer platform is provided for computer assisted navigation during surgery. The computer platform includes at least one processor that is operative to transform pre-operative images of a patient obtained from a first imaging modality to an estimate of the pre-operative images of the patient in a second imaging modality that is different than the first imaging modality. The at least one processor is further operative to register the estimate of the pre-operative images of the patient in the second imaging modality to intra-operative navigable images or data of the patient.
1. A method comprising:
transforming pre-operative images of a patient obtained from a first imaging modality to an estimate of the pre-operative images of the patient in a second imaging modality that is different than the first imaging modality; and
registering the estimate of the pre-operative images of the patient in the second imaging modality to intra-operative navigable images or data of the patient,
wherein the transforming of the pre-operative images of the patient obtained from the first imaging modality to the estimate of the pre-operative images of the patient in the second imaging modality, comprises:
encoding pre-operative image of the patient obtained from the first imaging modality to output a content vector indicating where anatomical features are located in the pre-operative images of the first imaging modality;
encoding pre-operative images of the patient obtained from the second imaging modality to output a style vector indicating how the anatomical features look in the pre-operative images of the second imaging modality; and
decoding the content vector indicating where the anatomical features are located in the pre-operative images of the first imaging modality using the style vector indicating how the anatomical features look in the pre-operative images of the second imaging modality; and
generating the estimate of the pre-operative images of the patient in the second imaging modality based on an output of the decoding.
2. The method of claim 1 , wherein the transforming of the pre-operative images of the patient obtained from the first imaging modality to the estimate of the pre-operative images of the patient in the second imaging modality, comprises:
processing the pre-operative images of the patient obtained from the first imaging modality through a neural networks model configured to transform pre-operative images in the first imaging modality to estimates of the pre-operative images in the second imaging modality, wherein the neural networks model has been trained based on matched sets of training images containing anatomical features captured by the first imaging modality and training images containing anatomical features captured by the second imaging modality, wherein at least some of the anatomical features captured by the first imaging modality correspond to at least some of the anatomical features captured by the second imaging modality.
3. The method of claim 2 , further comprising:
performing the training of the neural networks model based on matched sets of training images containing anatomical features captured by the first imaging modality and training images containing anatomical features captured by the second imaging modality.
4. The method of claim 1 , wherein:
the transforming of the pre-operative images of the patient obtained from the first imaging modality to the estimate of the pre-operative images of the patient in the second imaging modality, comprises transforming pre-operative magnetic resonance imaging (MRI) images of
the patient to synthetic x-ray images of the patient; and
the registering comprises registering the synthetic x-ray images of the patient to intraoperative navigable x-ray images of the patient, wherein the intra-operative navigable x-ray images are registered to a coordinate system of a camera tracking system.
5. The method of claim 4 , wherein the transforming of the pre-operative MRI images of the patient to the synthetic x-ray images of the patient, comprises:
transforming the pre-operative MRI images of the patient to synthetic computerized tomography (CT) images of the patient; and
transforming the synthetic CT images of the patient to the synthetic x-ray images.
6. The method of claim 5 , wherein the transforming of the pre-operative MRI images of the patient to the synthetic CT images of the patient, comprises:
processing the pre-operative MRI images of the patient through a neural networks model configured to transform pre-operative MRI images to synthetic CT images, wherein the neural networks model has been trained based on matched sets of training MRI images containing anatomical features captured by MRI modality and training CT images containing anatomical features captured by CT imaging modality, wherein at least some of the anatomical features captured by the MRI modality correspond to at least some of the anatomical features captured by the CT imaging modality.
7. The method of claim 5 , further comprising:
obtain a first slice set of pre-operative MRI images of the patient having higher resolution in a first plane and a lower resolution in a second plane orthogonal to the first plane;
obtain a second slice set of pre-operative MRI image slices of the patient having higher resolution in the second plane and a lower resolution in the first plane;
merging the first and second slice sets of pre-operative MRI images by registration of anatomical features captured in both of the the first and second slice sets of pre-operative MRI images, to output a merged slice set of pre-operative MRI images,
wherein the merged slice set of pre-operative MRI images are processed through the neural networks model for transform to the synthetic CT images.
8. The method of claim 1 , wherein:
the transforming of the pre-operative images of the patient obtained from the first imaging modality to the estimate of the pre-operative images of the patient in the second imaging modality, comprises transforming pre-operative magnetic resonance imaging (MRI) images or computerized tomography (CT) images of the patient to synthetic ultrasound images of the patient; and
the registering comprises registering the synthetic ultrasound images to intra-operative navigable ultrasound images of the patient, wherein the intra-operative navigable ultrasound images are registered to a coordinate system of a camera tracking system.
9. The method of claim 8 , wherein the transforming of the pre-operative magnetic resonance imaging (MRI) images or the computerized tomography (CT) images of the patient to the synthetic ultrasound images of the patient, comprises:
processing the pre-operative MRI images or CT images of the patient through a neural networks model configured to transform pre-operative MRI images or CT images to synthetic ultrasound images, wherein the neural networks model has been trained based on matched sets of: 1) training ultrasound images; and 2) either training MRI images or training CT images,
wherein the matched sets of: 1) training ultrasound images; and 2) either training MRI images or training CT images, have defined correspondences between anatomical features captured in images of the matched sets.
10. The method of claim 1 , wherein:
the transforming of the pre-operative images of the patient obtained from the first imaging modality to the estimate of the pre-operative images of the patient in the second imaging modality, comprises transforming pre-operative magnetic resonance imaging (MRI) images or computerized tomography (CT) images of the patient to synthetic optical camera images of the patient; and
the registering comprises registering the synthetic optical camera images to intraoperative navigable optical camera images of the patient, wherein the intra-operative navigable optical camera images are registered to a coordinate system of a camera tracking system.
11. The method of claim 10 , wherein the transforming of the pre-operative magnetic resonance imaging (MRI) images or computerized tomography (CT) images of the patient to the synthetic optical camera images of the patient, comprises:
processing the pre-operative MRI images or CT images of the patient through a neural networks model configured to transform pre-operative MRI images or CT images to synthetic optical camera images, wherein the neural networks model has been trained based on matched sets of: 1) training optical camera images; and 2) either training MRI images or training CT images,
wherein the matched sets of: 1) training optical camera images; and 2) either training MRI images or training CT images, have defined correspondences between anatomical features captured in images of the matched sets.
12. The method of claim 1 , wherein: the first and second imaging modalities are different ones of: magnetic resonance imaging (MRI) modality; computerized tomography (CT) imaging modality; and ultrasound imaging modality.
13. The method of claim 1 , wherein: the encoding of the pre-operative image of the patient obtained from the first imaging modality to output the content vector indicating where anatomical features are located in the preoperative images of the first imaging modality, comprises processing the pre-operative image of the patient in the first imaging modality through a first neural networks model configured to output the content vector indicating where anatomical features are located in the pre-operative images of the first imaging modality; the encoding of the pre-operative images of the patient obtained from the second imaging modality to output the style vector indicating how the anatomical features look in the preoperative images of the second imaging modality, comprises processing the pre-operative images of the patient obtained from the second imaging modality through a second neural networks model configured to output the style vector indicating how the anatomical features look in the pre-operative images of the second imaging modality; and the decoding of the content vector indicating where the anatomical features are located in the pre-operative images of the first imaging modality using the style vector indicating how the anatomical features look in the pre-operative images of the second imaging modality, comprises processing the content vector and the style vector through a third neural networks model configured to output the estimate of the pre-operative images of the patient in the second imaging modality.
14. The method of claim 13 , further comprising:
performing training of the first and second neural networks model,
wherein the training alternates between a training cycle using a style consistency loss operation to train based on differences in content between the pre-operative image from the first and second imaging modalities and another training cycle using a content consistency loss operation to train based on differences in style between the pre-operative image from the first and second imaging modalities.
15. The method of claim 13 , further comprising:
processing the estimate of the pre-operative images of the patient in the second imaging modality through a fourth neural networks model configured to detect keypoints in the preoperative images of the patient in the second imaging modality; and
tuning parameters of the second neural networks model based on the detected keypoints in the pre-operative images of the patient in the second imaging modality.
16. A computer platform for computer assisted navigation during surgery, comprising at least one processor operative to:
transform pre-operative images of a patient obtained from a first imaging modality to an estimate of the pre-operative images of the patient in a second imaging modality that is different than the first imaging modality; and
register the estimate of the pre-operative images of the patient in the second imaging modality to intra-operative navigable images or data of the patient, wherein the transformation of the preoperative images of the patient obtained from the first imaging modality to the estimate of the pre-operative images of the patient in the second imaging modality, comprises to:
encode pre-operative image of the patient obtained from the first imaging modality to output a content vector indicating where anatomical features are located in the pre-operative images of the first imaging modality;
encode pre-operative images of the patient obtained from the second imaging modality to output a style vector indicating how the anatomical features look in the pre-operative images of the second imaging modality; and
decode the content vector indicating where the anatomical features are located in the preoperative images of the first imaging modality using the style vector indicating how the anatomical features look in the pre-operative images of the second imaging modality; and
generate the estimate of the pre-operative images of the patient in the second imaging modality based on an output of the decoding.
17. The computer platform of claim 16 , wherein the transformation of the preoperative images of the patient obtained from the first imaging modality to the estimate of the pre-operative images of the patient in the second imaging modality, comprises to:
process the pre-operative images of the patient obtained from the first imaging modality through a neural networks model configured to transform pre-operative images in the first imaging modality to estimates of the pre-operative images in the second imaging modality, wherein the neural networks model has been trained based on matched sets of training images containing anatomical features captured by the first imaging modality and training images containing anatomical features captured by the second imaging modality, wherein at least some of the anatomical features captured by the first imaging modality correspond to at least some of the anatomical features captured by the second imaging modality.
18. The computer platform of claim 16 , wherein: the encoding of the pre-operative image of the patient obtained from the first imaging modality to output the content vector indicating where anatomical features are located in the preoperative images of the first imaging modality, comprises to process the pre-operative image of the patient in the first imaging modality through a first neural networks model configured to output the content vector indicating where anatomical features are located in the pre-operative images of the first imaging modality; the encoding of the pre-operative images of the patient obtained from the second imaging modality to output the style vector indicating how the anatomical features look in the preoperative images of the second imaging modality, comprises to process the pre-operative images of the patient obtained from the second imaging modality through a second neural networks model configured to output the style vector indicating how the anatomical features look in the pre-operative images of the second imaging modality; and the decoding of the content vector indicating where the anatomical features are located in the pre-operative images of the first imaging modality using the style vector indicating how the anatomical features look in the pre-operative images of the second imaging modality, comprises to process the content vector and the style vector through a third neural networks model configured to output the estimate of the pre-operative images of the patient in the second imaging modality.