Systems and methods for identifying and facilitating an intended interaction with a target object in a surgical space
An exemplary system includes a memory storing instructions and a processor communicatively coupled to the memory. The processor may be configured to execute the instructions to: detect an intent of a user of a computer-assisted surgical system to use a robotic instrument attached to the computer-assisted surgical system to interact with a target object while the target object is located in a surgical space; determine a pose of the target object in the surgical space; and perform, based on the detected intent of the user to interact with the target object and the determined pose of the target object in the surgical space, an operation with respect to the target object.
1 . A system comprising:
a memory storing instructions; and
a processor communicatively coupled to the memory and configured to execute the instructions to:
detect an intent of a user of a computer-assisted surgical system to use a robotic instrument attached to the computer-assisted surgical system to interact with a non-robotic instrument while the non-robotic instrument is located in a surgical space, wherein the non-robotic instrument is not directly attached to the robotic instrument but is movable within the surgical space by interaction with the robotic instrument;
determine a pose of the non-robotic instrument in the surgical space; and
perform, based on the detected intent of the user to interact with the non-robotic instrument and the determined pose of the non-robotic instrument in the surgical space, an operation with respect to the non-robotic instrument,
wherein the operation comprises providing an instruction to the user of the computer-assisted surgical system to assist the user in operating a user control console of the computer-assisted surgical system to instruct the computer-assisted surgical system to control the robotic instrument to move toward and engage with the non-robotic instrument, the instruction provided to the user prior to operation of the non-robotic instrument in the surgical space.
2 . The system of claim 1 , wherein:
the processor is further configured to execute the instructions to determine that the non-robotic instrument is located in the surgical space;
the detecting of the intent of the user to use the robotic instrument to interact with the non-robotic instrument is performed in response to the determining that the non-robotic instrument is located in the surgical space; and
the determining that the non-robotic instrument is located in the surgical space includes:
obtaining a depth map of the surgical space;
extracting a representation of an object from the depth map of the surgical space;
comparing the extracted representation of the object to a plurality of representations of known non-robotic instruments;
identifying, from the plurality of representations of known non-robotic instrument, a representation of a known non-robotic instrument that matches the extracted representation of the object; and
identifying the object as the non-robotic instrument based on the identified match.
3 . The system of claim 1 , wherein:
the non-robotic instrument includes a protrusion provided on an outer surface of the non-robotic instrument, the protrusion configured to be grasped by the robotic instrument attached to the computer-assisted surgical system; and
the determining of the pose of the non-robotic instrument includes determining a pose of the protrusion on the non-robotic instrument.
4 . The system of claim 1 , wherein the determining of the pose of the non-robotic instrument includes detecting an orientation of one or more markers provided on an outer surface of the non-robotic instrument.
5 . The system of claim 1 , wherein the determining of the pose of the non-robotic instrument includes:
accessing an image of the non-robotic instrument in the surgical space;
determining a representation of the non-robotic instrument from a viewpoint of an imaging device;
comparing the determined representation to a two-dimensional (2D) projection of a three-dimensional (3D) model of the non-robotic instrument that is oriented in a known orientation;
determining, based on the comparing of the determined representation to the 2D projection of the 3D model, a projection error between the determined representation and the projection of the 3D model; and
identifying the non-robotic instrument as being oriented in the known orientation when the determined projection error is less than a predefined threshold.
6 . The system of claim 1 , wherein the performing of the operation with respect to the non-robotic instrument further includes instructing the user of the computer-assisted surgical system to control the robotic instrument so as to grasp the non-robotic instrument with the robotic instrument.
7 . The system of claim 6 , wherein the processor is configured to provide at least one of visual guidance, audible guidance, or haptic feedback guidance to assist the user of the computer-assisted surgical system in controlling the robotic instrument to grasp the non-robotic instrument.
8 . The system of claim 1 , wherein:
the performing of the operation with respect to the non-robotic instrument further includes providing an instruction to the user of the computer-assisted surgical system to reposition the non-robotic instrument in the surgical space; and
the processor is further configured to execute the instructions to:
determine an updated pose of the non-robotic instrument in the surgical space; and
perform, based the determined updated pose of the non-robotic instrument in the surgical space, an additional operation with respect to the non-robotic instrument.
9 . The system of claim 1 , wherein the processor is further configured to execute the instructions to:
determine a position for the robotic instrument to assume to facilitate the robotic instrument interacting with the non-robotic instrument located in the surgical space; and
facilitate the robotic instrument assuming the position.
10 . The system of claim 9 , wherein:
the determining of the position for the robotic instrument to assume to facilitate the robotic instrument interacting with the non-robotic instrument includes determining a pose for the robotic instrument to assume to facilitate the robotic instrument interacting with the non-robotic instrument located in the surgical space; and
the facilitating of the robotic instrument assuming the position includes facilitating the robotic instrument assuming the pose.
11 . The system of claim 10 , wherein the determining of the pose for the robotic instrument to assume to facilitate the robotic instrument interacting with the non-robotic instrument located in the surgical space includes:
accessing a database that includes a set of candidate orientations for the robotic instrument to assume to facilitate the robotic instrument interacting with the non-robotic instrument;
selecting an orientation from the set of candidate orientations included in the database;
selecting a position for the robotic instrument to assume within the surgical space in relation to the non-robotic instrument; and
determining the pose based on the selected orientation and the selected position.
12 . The system of claim 10 , wherein the facilitating of the robotic instrument assuming the pose includes providing a representation of the robotic instrument in the determined pose for display to the user of the computer-assisted surgical system.
13 . The system of claim 10 , wherein the facilitating of the robotic instrument assuming the pose includes generating a motion path for the robotic instrument to follow to assume the pose.
14 . The system of claim 13 , wherein the generating of the motion path for the robotic instrument to follow to assume the pose includes:
determining that there is an obstruction in the generated motion path; and
performing an operation to facilitate removal of the obstruction from the generated motion path.
15 . The system of claim 13 , wherein:
the facilitating of the robotic instrument assuming the pose includes facilitating the robotic instrument following the motion path to assume the pose; and
the facilitating of the robotic instrument following the motion path to assume the pose includes facilitating a user of a computer-assisted surgical system attached to the robotic instrument moving the robotic instrument along the motion path.
16 . The system of claim 15 , wherein the facilitating of the user of the computer-assisted surgical system moving the robotic instrument along the motion path includes providing at least one of visual guidance, audible guidance, or haptic feedback guidance to assist the user in moving the robotic instrument along the motion path.
17 . The system of claim 10 , wherein the determined pose is a grasp pose that facilitates the robotic instrument grasping the non-robotic instrument.
18 . A system comprising:
a memory storing instructions; and
a processor communicatively coupled to the memory and configured to execute the instructions to:
determine that a non-robotic instrument is located in a surgical space, wherein the non-robotic instrument is not directly attached to a robotic instrument of a computer-assisted surgical system but is movable within the surgical space by interaction with the robotic instrument;
determine, in response to the determining that the non-robotic instrument is located in the surgical space, a pose of the non-robotic instrument in the surgical space; and
facilitate, based on the determined pose of the non-robotic instrument in the surgical space, the robotic instrument, which is attached to the computer-assisted surgical system, interacting with the non-robotic instrument,
wherein the facilitating of the robotic instrument interacting with the non-robotic instrument comprises providing an instruction to a user of a computer-assisted surgical system to assist the user in operating a user control console of the computer-assisted surgical system to instruct the computer-assisted surgical system to control the robotic instrument to move toward and engage with the non-robotic instrument, the instruction provided to the user prior to operation of the non-robotic instrument in the surgical space.
19 . A method comprising:
detecting, by a processor associated with a computer-assisted surgical system, an intent of a user of the computer-assisted surgical system to use a robotic instrument attached to the computer-assisted surgical system to interact with a non-robotic instrument while the non-robotic instrument is located in a surgical space, wherein the non-robotic instrument is not directly attached to the robotic instrument but is movable within the surgical space by interaction with the robotic instrument;
determining, by the processor, a pose of the non-robotic instrument in the surgical space; and
performing, by the processor based on the detected intent of the user to interact with the non-robotic instrument and the determined pose of the non-robotic instrument in the surgical space, an operation with respect to the non-robotic instrument,
wherein the operation comprises providing an instruction to the user of the computer-assisted surgical system to assist the user in operating a user control console of the computer-assisted surgical system to instruct the computer-assisted surgical system to control the robotic instrument to move toward and engage with the non-robotic instrument, the instruction provided to the user prior to operation of the non-robotic instrument in the surgical space.