Methods of locating and tracking robotic instruments in robotic surgical systems

1. A method to locate a robotic instrument in a field of view of a camera, the method comprising:

capturing sequential images in a field of view of a camera, the sequential images being correlated between successive views;

receiving a kinematic datum to provide an approximate location of the robotic instrument; and

analyzing the sequential images in response to the approximate location of the robotic instrument.

2. The method of claim 1 , further comprising:

locating the robotic instrument in the field of view in response to the analysis of the sequential images.

3. The method of claim 1 , further comprising:

estimating a current location of the robotic instrument in the field of view in response to the analysis of the sequential images and a portion of the robotic instrument being hidden from the field of view.

4. The method of claim 1 , further comprising:

synthesizing a model of the robotic instrument; and

further analyzing the sequential images in response to the model of the robotic instrument.

5. The method of claim 1 , wherein

the kinematic datum is a current kinematic datum, and the method further comprises

correcting the current kinematic datum to correct future kinematic datum without regard to corrections of past kinematic datum.

6. The method of claim 1 , wherein

the kinematic datum is a current kinematic datum, and the method further comprises

correcting the current kinematic datum in response to corrections of past kinematic datum such that future kinematic datum can be corrected.

7. The method of claim 1 , wherein

the sequential images form a depth map, and

the depth map is analyzed in response to the approximate location of the robotic instrument to locate the robotic instrument in the field of view.

8. A method for robotic systems, the method comprising:

receiving one or more video images of at least one robotic instrument in reference to a camera coordinate system;

receiving kinematics information associated with the at least one robotic instrument; and

localizing the at least one robotic instrument in the camera coordinate system in response to the one or more video images and the kinematics information.

9. The method of claim 8 , wherein

the localizing of the at least one robotic instrument includes

predicting the position of the at least one robotic instrument in response to the kinematics information.

10. The method of claim 9 , wherein

the localizing of the at least one robotic instrument includes

synthesizing an image of the at least one robotic instrument in response to a computer aided design (CAD) model of the at least one robotic instrument; and

analyzing the video images to match the synthesized tool image to an actual tool image to determine a position of the robotic instrument.

11. The method of claim 10 , wherein

the analyzing to determine the position includes

minimizing a cost function of comparing the synthesized tool image with the actual tool image, and

minimizing a sum of squared differences between geometries of the synthesized tool image and the actual tool image.

12. The method of claim 11 , wherein

the minimizing of the sum of squared differences includes

non-linearly mapping a three dimensional synthesized tool image into a two dimensional synthesized tool image.

13. The method of claim 12 , further comprising:

learning a change in appearance of the at least one robotic instrument in response to a different view of the at least one robotic instrument.

14. The method of claim 13 , wherein

the change in appearance includes a change in one or more of lighting, occlusion, wear, and the surrounding environment of the robotic instrument.

15. The method of claim 8 , further comprising:

after localizing the at least one robotic instrument,

tracking the motion of the at least one robotic instrument in the camera coordinate system in response to the one or more video images and the kinematics information.

16. The method of claim 15 , wherein

if while tracking the motion the at least one instrument is occluded so as to become untrackable, then

the localizing of the at least one robotic instrument in the camera coordinate system is repeated to locate the robotic instrument in the camera coordinate system.

17. The method of claim 15 , wherein

the tracking of the motion of the at least one robotic instrument includes

adaptively fusing the video images and the kinematics information together to determine a position of the at least one robotic instrument.

18. The method of claim 17 , wherein

the adaptively fusing includes

weighting the kinematics information more than the video images if the video images are unreliable indicators of a location of the at least one robotic instrument.

19. The method of claim 17 , wherein

the adaptively fusing includes

weighting the video images more than kinematics information if the kinematics information are unreliable indicators of a location of the at least one robotic instrument.

20. The method of claim 17 , wherein

the adaptively fusing includes

weighting the video images more than kinematics information if the video images are reliable indicators of a location of the at least one robotic instrument.

21. The method of claim 17 , wherein

the adaptively fusing includes

using a covariance matrix with a state space model to adaptively fuse the video images and kinematics information together to provide a reliable indicator of a location of the at least one robotic instrument.

22. A method for indicating tool entrance into a field of view of a camera, the method comprising:

tracking a robotic surgical tool in and out of a field of view of a camera;

determining that the robotic surgical tool is outside the field of view of the camera in response to the tracking of the robotic surgical tool; and

displaying one of a plurality of compass icons in a display of the field of view to indicate a direction of tool reentrance into the field of view.

23. The method of claim 22 , further comprising:

altering the one compass icon in the display by flashing, adding arrowheads, or changing color to indicate an increase and a decrease in a distance between an edge of the field of view of the camera and the robotic surgical tool.

24. The method of claim 22 , wherein

the camera has moved away from the position of the robotic surgical tool in a surgical site such that the robotic surgical tool is outside the field of view of the camera.

25. The method of claim 22 , wherein

the robotic surgical tool has moved away from a position of the camera in a surgical site such that the robotic surgical tool is outside the field of view of the camera.