Robotic surgical safety via video processing
One example method for improving robotic surgical safety via video processing includes identifying, during a robotic surgical procedure, one or more surgical tools using one or more images of the surgical procedure captured by a camera, the robotic surgical procedure employing a robotic surgical device controlling the one or more identified surgical tools; predicting, for at least one of the one or more images, one or more loaded surgical tools that are controlled by the robotic surgical device and should be in a field of view of the camera; comparing the one or more identified surgical tools with the one or more loaded surgical tools; determining that at least one of the one or more loaded surgical tools does not match any of the one or more identified surgical tools; and causing the at least one loaded tool of the robotic surgical device to be disabled.
1. A method that includes one or more processing devices performing operations comprising:
identifying a surgical tool used by a robotic surgical device in a surgical procedure using one or more images of the surgical procedure;
estimating a pose of the surgical tool based, at least in part, upon the one or more images;
determining a difference between the estimated pose of the surgical tool and a predicted pose of the surgical tool predicted through a kinematic chain model of the robotic surgical device;
calculating a force exerted by the surgical tool based, at least in part, upon the difference of the estimated pose and the predicted pose of the surgical tool;
determining that the force is higher than a threshold; and
causing the force of the surgical tool to be reduced.
2. The method of claim 1 , wherein estimating the pose of the surgical tool based on the one or more images comprises at least one of
matching an image of the surgical tool with a database of tool pose images, or
predicting the pose of the surgical tool by applying a neural network trained to identify tool poses on the one or more images.
3. The method of claim 1 , wherein estimating the pose of the surgical tool comprises estimating a depth of the surgical tool relative to a camera capturing the one or more images.
4. The method of claim 3 , wherein the depth of the surgical tool is estimated based on one or more of a geometry of the surgical tool, one or more fiducials on the surgical tool, or one or more stereo images of the surgical tool.
5. The method of claim 1 , wherein the force of the surgical tool is calculated further based on one or more of a geometry of the surgical tool or a material of the surgical tool.
6. The method of claim 1 , wherein identifying the surgical tool using the one or more images comprises:
estimating a position of the surgical tool; and
recognizing a type of the surgical tool using the one or more images.
7. The method of claim 1 , wherein identifying the surgical tool from the one or more images comprises:
identifying an identifier visible from the one or more images; and
interpreting the identifier to identify the surgical tool.