Methods and systems for using multi view pose estimation
The present invention is disclosing several methods related to intra-body navigation of radiopaque instrument through natural body cavities. One of the methods is disclosing the pose estimation of the imaging device using multiple images of radiopaque instrument acquired in the different poses of imaging device and previously acquired imaging. The other method allows to resolve the radiopaque instrument localization ambiguity using several approaches, such as radiopaque markers and instrument trajectory tracking.
1. A method, comprising:
obtaining a first image from a first imaging modality,
extracting at least one element from the first image from the first imaging modality, wherein the at least one element comprises an airway, a blood vessel, a body cavity, or any combination thereof;
obtaining, from a second imaging modality, at least (i) a first image of a radiopaque instrument in a first pose of second imaging modality and (ii) a second image of the radiopaque instrument in a second pose of second imaging modality,
wherein the radiopaque instrument is in a body cavity of a patient;
generating at least two augmented bronchograms,
wherein a first augmented bronchogram corresponds to the first image of the second imaging modality in the first pose, and
wherein a second augmented bronchogram corresponds to the second image of the second imaging modality in the second pose,
determining mutual geometric constraints between:
(i) the first pose of the of second imaging modality, and
(ii) the second pose of the of second imaging modality,
estimating the first pose of the of second imaging modality and the second pose of the of second imaging modality,
wherein the estimation is performed using:
(i) the first augmented bronchogram,
(ii) the second augmented bronchogram, and
(iii) the at least one element, and
wherein the estimated first pose of the of second imaging modality and the estimated second pose of the of second imaging modality meets the determined mutual geometric constraints,
generating a third image; wherein the third image is an augmented image derived from the second imaging modality which highlights an area of interest,
wherein the area of interest is determined from projecting data from the estimated first pose and the estimated second pose.
2. The method of claim 1 , wherein there is additional element from the first image from the first imaging modality and second image of second imaging modality further comprises a rib, a vertebra, a diaphragm, or any combination thereof.
3. The method of claim 1 , wherein the mutual geometric constraints are generated by:
a. estimating a difference between (i) the first pose and (ii) the second pose by comparing the first image of the radiopaque instrument and the second image of the radiopaque instrument,
wherein the estimating is performed using a device comprising a protractor, an accelerometer, a gyroscope, or any combination thereof, and wherein the device is attached to the second imaging modality;
b. extracting a plurality of image features to estimate a relative pose change,
wherein the plurality of image features comprise anatomical elements, non-anatomical elements, or any combination thereof,
wherein the image features comprise: patches attached to a patient, radiopaque markers positioned in a field of view of the second imaging modality, or any combination thereof,
wherein the image features are visible on the first image of the radiopaque instrument and the second image of the radiopaque instrument;
c. estimating a difference between (i) the first pose and (ii) the second pose by using a at least one camera,
wherein the camera comprises: a video camera, an infrared camera, a depth camera, or any combination thereof,
wherein the camera is at a fixed location,
wherein the camera is configured to track at least one feature, wherein the at least one feature comprises: a marker attached the patient, a marker attached to the second imaging modality, or any combination thereof, and
tracking the at least one feature;
d. or any combination thereof.
4. The method of claim 1 , further comprising: tracking the radiopaque instrument for:
identifying a trajectory, and
using the trajectory as a further geometric constraint,
wherein the radiopaque instrument comprises an endoscope, an endo-bronchial tool, or a robotic arm.
5. The method of claim 1 , wherein the first image from the first imaging modality is a pre-operative image.
6. The method of claim 1 , wherein the at least one image of the radiopaque instrument from the second imaging modality is an intra-operative image.