Apparatus and method for tracking hand-held surgical tools

1 . A surgical method, comprising:

capturing, using a first image capture device, first image data that includes both a first image portion indicating a first optical target residing on a hand-held surgical tool and a second image portion indicating a second optical target proximate to a patient,

wherein the first optical target is located on the hand-held surgical tool at a first predefined target location,

wherein the second optical target is located on the patient at a second predefined target location, and

wherein a location relationship between a tissue of interest of the patient and the second predefined location of the second optical target is known;

determining a location of the first optical target and an orientation of the first optical target in a three dimensional (3D) space based on the first image portion of the first image data received only from the first image capture device;

retrieving hand-held surgical tool model data that represents the hand-held surgical tool, wherein the hand-held surgical tool model data comprises first optical target location model data that corresponds to the predefined location of the first optical target;

determining a current location of the hand-held surgical tool and a current orientation of the hand-held surgical tool in the 3D space based on the determined location of the first optical target and the determined orientation of the first optical target;

determining a location of the second optical target in the 3D space based on the second image portion of the first image data received only from the first image capture device; and

determining a location of the tissue of interest in the 3D space based on the determined location of the second optical target on the patient.

2 . The surgical method of claim 1 , further comprising:

accessing tissue model data defining a tissue model corresponding to the tissue of interest of the patient,

wherein the tissue model was previously generated from an examination of the patient prior to the current surgical procedure.

3 . The surgical method of claim 2 ,

wherein the tissue model data was previously generated based on a series of acquired time indexed sonogram images.

4 . The surgical method of claim 2 , further comprising:

generating a composite image,

wherein the generated composite image concurrently presents a first graphical representation of the tissue of interest of the patient based on the tissue model and a second graphical representation of the hand-held surgical tool; and

presenting the generated composite image on a display.

5 . The surgical method of claim 4 ,

wherein the hand-held surgical tool model data includes working end location model data that corresponds to a location of a working end of the hand-held surgical tool, and

wherein the working end of the hand-held surgical tool is disposed on a distal end of the hand-held surgical tool,

the method further comprising:

determining a location of the working end of the hand-held surgical tool and an orientation of the working end of the hand-held surgical tool based on the hand-held surgical tool model data.

6 . The surgical method of claim 4 , wherein the captured first image data is one of a plurality of time sequenced captured images of a video stream captured by the image capture device, the method further comprising:

continuously updating the generated image that is presented on the display using currently acquired first image data that is received in the video stream,

wherein the surgical team member viewing the display intuitively understands the current location of the hand-held surgical tool and the current orientation of the hand-held surgical tool with respect to the tissue of interest of the patient in real-time.

7 . The surgical method of claim 2 ,

wherein the image capture device is a video camera.

8 . The surgical method of claim 1 ,

wherein a third optical target is located on the hand-held surgical tool at a second predefined target location that is different from the first predefined target location,

wherein the hand-held surgical tool model data comprises third optical target location model data that corresponds to the predefined location of the third optical target,

the method further comprising:

determining a location of the third optical target the 3D space based only on the first image data, and based on the hand-held surgical tool model data,

wherein determining the current location of the hand-held surgical tool and the current orientation of the hand-held surgical tool in the 3D space is based on the determined location of the first optical target and the determined location of the third optical target.

9 . The surgical method of claim 1 ,

wherein the first optical target is on a first surgical tool member of the hand-held surgical tool at the first predefined target location on the first surgical tool member of the hand-held surgical tool,

wherein the hand-held surgical tool includes a second surgical tool member that is coupled to the first surgical tool member,

wherein a third optical target is located on the second member of the hand-held surgical tool at a third predefined target location, and

wherein the hand-held surgical tool model data further comprises third optical target location model data that corresponds to the predefined location of the third optical target,

the method further comprising:

determining a current location of the first member of the hand-held surgical tool and a current orientation of the first member of the hand-held surgical tool in the 3D space based on the determined location of the first optical target and the determined orientation of the first optical target;

determining a location of the third optical target and an orientation of the third optical target in the 3D space based on the first image data, and based on the hand-held surgical tool model data; and

determining a current location of the second member of the hand-held surgical tool and a current orientation of the second member of the hand-held surgical tool in the 3D space based on the determined location of the third optical target and the determined orientation of the third optical target.

10 . The surgical method of claim 9 ,

wherein a first surgical tool member is disposed on a distal end of the first surgical tool member,

wherein a second surgical tool member is disposed on a distal end of the second surgical tool member,

wherein the first surgical tool member and the second surgical tool member cooperatively operate in a coordinated manner to perform a surgical operation on the tissue of the patient, and

wherein the method further comprises:

determining a current location of the first surgical tool member in the 3D space based on the determined location and orientation of the first surgical tool member, and based on the hand-held surgical tool model data; and

determining a current location of the second surgical tool member in the 3D space based on the determined location and orientation of the second surgical tool member, and based on the hand-held surgical tool model data.

11 . The surgical method of claim 1 ,

wherein the first optical target is on a first surgical tool member of the hand-held surgical tool at the first predefined target location on the first surgical tool member of the hand-held surgical tool,

wherein a third optical target is on a first surgical tool member of the hand-held surgical tool at a third predefined target location on the first surgical tool member that is different from the first predefined target location,

wherein the hand-held surgical tool includes a second surgical tool member that is coupled to the first surgical tool member,

wherein a fourth optical target is located on the second member of the hand-held surgical tool at a fourth predefined target location,

wherein a fifth optical target is on a second surgical tool member of the hand-held surgical tool at a fifth predefined target location on the first surgical tool member that is different from the fourth predefined target location,

wherein the hand-held surgical tool model data further comprises:

third optical target location model data that corresponds to the predefined location of the third optical target;

fourth optical target location model data that corresponds to the predefined location of the fourth optical target; and

fifth optical target location model data that corresponds to the predefined location of the fifth optical target,

the method further comprising:

determining a location of the third optical target in the 3D space based only on the first image data, and based on the hand-held surgical tool model data;

determining a location of the fourth optical target in the 3D space based only on the first image data, and based on the hand-held surgical tool model data

determining a location of the fifth optical target in the 3D space based on the first image data, and based on the hand-held surgical tool model data;

determining a current location of the first member of the hand-held surgical tool and a current orientation of the first member of the hand-held surgical tool in the 3D space based on the determined location of the first optical target and the determined location of the third optical target, and based on the hand-held surgical tool model data; and

determining a current location of the second member of the hand-held surgical tool and a current orientation of the second member of the hand-held surgical tool in the 3D space based on the determined location of the fourth optical target and the determined location of the fifth optical target, and based on the hand-held surgical tool model data.

12 . The surgical method of claim 1 ,

wherein the first optical target is on a first surgical tool member of the hand-held surgical tool at the first predefined target location on the first surgical tool member of the hand-held surgical tool,

wherein the hand-held surgical tool includes a second surgical tool member that is coupled to the first surgical tool member,

wherein a third optical target is located on the second member of the hand-held surgical tool at a third predefined target location,

wherein a fourth first optical target is located where the first surgical tool member is coupled to the second surgical tool member,

wherein the hand-held surgical tool model data further comprises:

third optical target location model data that corresponds to the predefined location of the third optical target; and

fourth optical target location model data that corresponds to the predefined location of the fourth optical target,

the method further comprising:

determining a location of the third optical target in the 3D space based on the first image data, and based on the hand-held surgical tool model data;

determining a location of the fourth optical target in the 3D space based on the first image data, and based on the hand-held surgical tool model data;

determining a current location of the first member of the hand-held surgical tool and a current orientation of the first member of the hand-held surgical tool in the 3D space based on the determined location of the first optical target and the determined location of the fourth optical target, and based on the hand-held surgical tool model data; and

determining a current location of the second member of the hand-held surgical tool and a current orientation of the second member of the hand-held surgical tool in the 3D space based on the determined location of the third optical target and the determined location of the fourth optical target, and based on the hand-held surgical tool model data.

13 . The surgical method of claim 12 ,

wherein a first surgical tool member is disposed on a distal end of the first surgical tool member

wherein a second surgical tool member is disposed on a distal end of the second surgical tool member,

wherein the first surgical tool member and the second surgical tool member cooperatively operate in a coordinated manner to perform a surgical operation on the tissue of the patient, and

wherein the method further comprises:

determining a current location of the first surgical tool member and a current orientation of the first surgical tool member in the 3D space based on determined location and orientation of the first surgical tool member, and based on the hand-held surgical tool model data; and

determining a current location of the second surgical tool member and a current orientation of the first surgical tool member in the 3D space based on determined location and orientation of the second surgical tool member, and based on the hand-held surgical tool model data.

14 . The surgical method of claim 13 ,

wherein the hand-held surgical tool is a surgical scissor,

wherein the first surgical tool member is a first cutting edge of the surgical scissor, and

wherein the second surgical tool member is a second cutting edge of the surgical scissor.

15 . The surgical method of claim 13 ,

wherein the manually manipulated surgical instrument is a surgical clamp,

wherein the first surgical tool member is a first clamp of the surgical clamp, and

wherein the second surgical tool member is a second clamp of the surgical clamp.

16 . The surgical method of claim 1 , wherein the hand-held surgical tool is a surgical scalpel with a surgical tool member that is a scalpel blade that is located at the distal end of the surgical scalpel.

17 . The surgical method of claim 1 , wherein the hand-held surgical tool is a syringe with a surgical tool member that is a needle located at a distal end of the syringe.

18 . The surgical method of claim 1 ,

wherein the hand-held surgical tool is a syringe with a surgical tool member that is a needle located at a distal end of the syringe,

wherein the first optical target is on a plunger of the syringe,

wherein a second optical target is on a barrel of the syringe, and

wherein the hand-held surgical tool model data further comprises:

second optical target location model data that corresponds to the predefined location of the second optical target; and

volume data corresponding to a volume of the barrel of the syringe,

the method further comprising:

determining a change in location of the first optical target with respect to the location of the first optical target;

determining a change in a travel distance of a barrel seal located at the distal end of the plunger, wherein the travel distance equal a distance in the change of the location of the first optical target relative to the second optical target; and

determining a volume defined within the barrel based on the determined travel distance of the barrel seal.

19 . The surgical method of claim 1 ,

wherein the surgical team member who is manually manipulating the manually manipulated surgical instrument is a first surgical team member that is viewing the rendered image on a first display,

wherein a second surgical team member is controlling operation of a robotic surgery system that robotically manipulates a robotic surgical instrument,

wherein the rendered image includes the first graphical representation of the tissue of interest of the patient, the second graphical representation of the manually manipulated surgical instrument being manipulated by the first surgical team member, and a third graphical representation of the robotic surgical instrument,

wherein the method further comprises:

presenting the rendered image on the first display that is being viewed by the first surgical team member; and

presenting the rendered image on a second display that is being viewed by the second surgical team member, and

wherein the rendered image is concurrently presented on the first display to the first surgical team member and on the second display to the second surgical team member.

20 . The surgical method of claim 19 , further comprising:

receiving robotic surgical instrument model information from the robotic surgery system;

receiving robotic surgical instrument location information from the robotic surgery system that is based on a current location of the robotic surgical instrument; and

receiving robotic surgical instrument orientation information from the robotic surgery system that is based on a current orientation of the robotic surgical instrument,

wherein the rendered image, when presented on the first display and the second display, indicates the location and the orientation of the robotic surgical instrument relative to the location and the orientation of the manually manipulated surgical instrument, and indicates the location and the orientation of the robotic surgical instrument relative to the organ of interest.