Devices, systems and methods for natural feature tracking of surgical tools and other objects

1. A system comprising:

a sensor comprising an optical sensor, the optical sensor configured to generate an optical sensor image of a tool and a target attached to the tool at a known positional relationship, the tool having an effector with the tool lying within a field of view of the optical sensor, the sensor configured to use the optical sensor image of the tool and to generate optical measurements of the target, wherein the target lying in the field of view of the optical sensor and providing positional information for the tool; and

a computing unit in communication with the sensor, the computing unit configured to:

calculate a pose of the target in up to six degrees of freedom using the optical measurements;

determine an expected location of the effector of the tool based on first pre-loaded information of the tool comprising a location for the effector on the tool responsive to a known position of the target on the tool;

determine a feature tracked location of the effector of the tool based on second pre-loaded information of the tool comprising feature information describing features of the tool for detection; and the features of the tool as detected from the optical sensor image of the tool;

wherein the expected location and the feature tracked location comprise two real-time location measures of the effector of the tool when located at a same location, the two measures determined in two different manners;

calculate a difference between the feature tracked location and expected location of the effector of the tool;

generate a confidence metric using the difference;

and provide positional measurements of the effector of the tool with respect to the optical sensor, the positional measurements being provided with the confidence metric and being responsive to the pose of the target.

2. The system of claim 1 , wherein the optical sensor is attached to an object which is different from the tool.

3. The system of claim 2 , wherein the object is a patient.

4. The system of claim 1 , further configured to provide the confidence metric to a display unit to display in one of a numerical or graphical format.

5. The system of claim 1 , wherein the computing unit is further configured to prevent surgical navigation using the positional measurements when the confidence metric is outside of a tolerance range.

6. The system of claim 1 , wherein the computing unit determines the feature tracked location of the effector of the tool by calculating a pose of the effector of the tool and further determines the expected location of the effector of the tool by calculating an expected pose of the effector of the tool.

7. The system of claim 1 , wherein the optical sensor image comprises a two-dimensional optical sensor image and the computing unit determines the feature tracked location of the effector of the tool by calculating a position of the effector of the tool within a coordinate frame of the two-dimensional optical sensor image and further determines the expected location of the effector of the tool by calculating an expected position of the effector of the tool within the coordinate frame of the two-dimensional optical sensor image.

8. A system to provide surgical navigation of an effector of a tool with respect to a pose of a target comprising:

a sensor comprising an optical sensor including a camera to generate an optical sensor image of a tool and a target attached to the tool at a known positional relationship, the tool having an effector with the tool lying within a field of view of the optical sensor for inclusion in optical sensor images by the camera, the sensor configured to use the optical sensor image of the tool and to generate optical measurements of the target, wherein the target lying in the field of view of the optical sensor and providing positional information for the tool is attached, the object being different from the tool; and

a computing unit in communication with the sensor, the computing unit configured to, when the optical sensor is attached to the tool:

calculate a pose of the target in up to six degrees of freedom using the optical measurements;

determine an expected location of the tool based on first pre-loaded information of the tool comprising a location for the effector on the tool responsive to a known position of the target on the tool;

generate a virtual tool projection based on the expected location of the tool;

generate a composite image comprising the optical sensor image and the virtual tool projection; and

provide the composite image to a display unit.

9. The system of claim 8 further comprising a display unit to display the composite image.

10. The system of claim 8 , wherein the computing unit is configured to generate the composite image further comprising virtual error bounds.

11. The system of claim 8 , wherein the optical sensor is attached to an object and wherein the optical sensor is configured to provide positional information of the target and wherein the computing unit is configured to provide surgical navigation with respect to the optical sensor.

12. The system of claim 11 , wherein the optical sensor is configured to attach to an anatomy of a patient.

13. The system of claim 8 , wherein the target further comprises a kinematic mount for attachment to a cooperating kinematic mount on the tool, and the pre-loaded information comprises a first positional relationship between the target and the kinematic mount of the sensor, and a second positional relationship between the cooperating kinematic mount and the tool.

14. The system of claim 8 , wherein an optically detectable marker is attached to the tool, and the computing unit is further configured to determine the expected location of the tool based on the pre-loaded information of the tool, the pre-loaded information comprising a spatial relationship between the target and the optically detectable marker, and the virtual tool projection comprising a virtual projection of the optically detectable marker.

15. A system comprising:

a sensor comprising an optical sensor, the optical sensor configured to generate an optical sensor image of a robot and of a target each lying in a field of view of the optical sensor, wherein:

one of the sensor and the target is configured to attach to the robot at a known positional relationship and the other of the sensor and the tool is configured to attach to an object that is different from the robot,

the robot comprising an effector,

the target providing positional information for one of the robot and the object to which the target is attached, and

the sensor is configured to use the optical sensor image of the robot and to generate optical measurements of the target; and

a computing unit in communication with the sensor, the computing unit configured to:

calculate a pose for the target in up to six degrees of freedom using the optical measurements, when the target is attached to the one of the robot and the object;

determine an expected location of the effector of the robot based on first pre-loaded information of the robot comprising a location for the effector on the robot responsive to a known position of the one of the sensor and target when attached to the robot;

determine a feature tracked location of the effector of the robot based on second pre-loaded information of the tool comprising feature information describing features of the robot for detection; and the features of the robot as detected from the optical sensor image of the tool;

wherein the expected location and the feature tracked location comprise two real-time location measures of the effector of the robot when located at a same location, the two measures determined in two different manners;

calculate a difference between the feature tracked location and expected location of the effector of the robot;

responsive to the difference as calculated, provide a signal to control a robotic system operating the robot.

16. The system of claim 15 , wherein the signal indicates to the robotic system to stop proceeding with a planned trajectory for the effector.

17. The system of claim 15 , wherein the computing unit is configured to calculate a confidence metric responsive to the difference and providing the signal responsive to the confidence metric.

18. The system of claim 15 , wherein the computing unit is further configured to provide positional measurements of the effector.