Surgical assistant system based on image data of the operative field
A method and system for assisting a physician compute the 3D location of the tip of a surgical device inserted into a patient is described. A trained model computes the 3D location of the hidden portion of the surgical device based on live image data of the operative field. A display shows the tip of the surgical device and a 3D model of the body organ in a fused arrangement.
1 . A system for assisting a physician compute the 3D location of a tip of a surgical device inserted into an organ of a patient comprising:
a plurality of cameras arranged outside of the patient to obtain operative-field image data; and
a processor programmed and operable to compute the 3D location of the device tip based on the operative-field image data; and
wherein the processor is further programmed and operable to evaluate and weight operative-field image data of each of the plurality of cameras for reliability, and to base the 3D location computation on the weighted operative-field image data; and
wherein evaluating and weighting the operative-field image data of each of the plurality of cameras for reliability comprises determining (a) the number of objects present in each camera's FOV, (b) the number of markers present in each camera's FOV, (c) the number of features present in each camera's FOV, and/or (d) the unobstructed area of objects in each camera's FOV.
2 . The system as recited in claim 1 , further comprising a detection module and classification module for, respectively, detecting and classifying objects in the operative field.
3 . The system as recited in claim 2 , wherein the objects are selected from the group comprising a bronchoscope, the surgical device, markers applied to the patient, bronchoscope or surgical device, patient and physician's anatomy.
4 . The system as recited in claim 3 , wherein the processor is operable to extract features from the objects, and the 3D location computation is based on the evaluating the extracted features from the objects.
5 . The system as recited in claim 4 , wherein the processor is further programmed and operable to compute a motion signature based on tracking one the extracted features over time, and the 3D location computation is based on evaluating the motion signature.
6 . The system as recited in claim 5 , wherein the object is an ablation device, and the feature is a portion of a handle of the ablation device.
7 . The system as recited in claim 4 , wherein the 3D location computation is further based on user input.
8 . The system as recited in claim 4 , further comprising an external camera tracking sensor for computing the location of the cameras, and wherein the 3D location computation is further based on tracking the location of the plurality of cameras during the procedure.
9 . The system of claim 1 , wherein one or more of the plurality of cameras are visible light spectrum cameras.
10 . The system as recited in claim 1 , wherein the 3D location computation is further based on live fluoroscopic image data and tracking the location of the fluoroscopic camera.
11 . The system as recited in claim 1 , further comprising a library of prior device data comprising a plurality of different device profiles, and wherein the 3D location computation is based on a device profile from the library.
12 . The system as recited in claim 2 , wherein the processor is further programmed and operable to transpose the operative field image data to a common perspective view prior to the detecting.
13 . A method for assisting a physician perform a medical procedure on a patient with tracking and guidance of a medical device comprises:
advancing at least portion of the medical device into an organ of the patient;
locating a hidden portion of the medical device in the organ of the patient based on live camera images of the operative field; and,
prior to the locating, selecting a device-trained model from a library of different device-trained models based on the type of medical device being advanced into the organ of the patient.
14 . The method as recited in claim 13 , wherein the step of locating comprises locating a tip of a bronchoscope, and locating a portion of a medical device advanced through the bronchoscope.
15 . The method as recited in claim 13 , wherein the locating is based on at least one visible marker applied to an unburied portion of the medical device, anatomy of the patient, or anatomy of the physician.
16 . The method as recited in claim 13 , further comprising extracting at least one feature from the object, and computing a motion signature of the extracted feature based on tracking the extracted feature over time, and wherein the locating step is based on the motion signature.
17 . A method for assisting a physician perform a medical procedure on a patient with tracking and guidance of a medical device comprises:
locating a hidden portion of the medical device in an organ of the patient based on live camera images of the operative field; and
evaluating and weighting operative-field image data of each of the plurality of cameras for reliability, and basing the locating on the weighted operative-field image data, and wherein the evaluating and weighting operative-field image data of each of the plurality of cameras for reliability comprises determining (a) the number of objects present in each camera's FOV, (b) the number of markers present in each camera's FOV, (c) the number of features present in each camera's FOV, and/or (d) the unobstructed area of objects in each camera's FOV.
18 . The method as recited in claim 17 , wherein the step of locating comprises locating a tip of a bronchoscope, and locating a portion of a medical device advanced through the bronchoscope.
19 . The method as recited in claim 17 , wherein the locating is based on at least one visible marker applied to an unburied portion of the medical device, anatomy of the patient, or anatomy of the physician.