Systems, methods, and computer-readable program products for controlling a robotically delivered manipulator
Systems, methods, and computer-readable media are provided for controlling a robotically delivered manipulator. One system includes a robotic manipulator having a base and a surgical instrument holder configured to move relative to the base, a surgical instrument removably coupled to the surgical instrument holder, a user interface configured to present information related to least one of the robotic manipulator or the surgical instrument, a gesture detection sensor configured to detect a gesture made by a user representing a desired movement of the robotic manipulator, and a controller configured to actuate the robotic manipulator in a predetermined manner corresponding to the detected gesture.
1 . A robotic surgical system comprising:
one or more robotic manipulators, each having a surgical instrument holder;
a surgical instrument removably coupled to the surgical instrument holder;
a user interface configured to present information related to at least one of the one or more robotic manipulators, the surgical instrument, or a surgical environment within which the surgical instrument is present, wherein the information includes a map indicating a position of the one or more robotic manipulators and surgical instruments within the surgical environment;
a gesture detection sensor configured to detect a gesture made by at least one of a hand of a user or a head of a user; and
a controller in communication with the one or more robotic manipulators, the surgical instrument, the user interface, and the gesture detection sensor and including one or more processors and one or more memories having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to:
determine whether or not a gesture indicating a selection of a first robotic manipulator of the one or more robotic manipulators has been detected;
provide one or more commands to actuate the first robotic manipulator in a predetermined manner corresponding to the detected gesture;
obtain position data including a first position and a second position of the detected gesture relative to a location on the map;
determine whether a difference between a first distance from the first position to the location and a second distance from the second position to the location is outside of a predetermined threshold range to indicate that the detected gesture corresponds to a selection of at least one of the one or more robotic manipulators or the surgical instruments; and
identify a gesture path of the detected gesture, the gesture path being within a proximity of at least one of a graphical representation of the surgical instrument or the one or more robotic manipulators on the map;
wherein, when it is determined that the gesture indicating selection of the first robotic manipulator has been detected, the one or more commands include retracting the surgical instrument holder supporting the first robotic manipulator from the surgical environment in order to decouple the surgical instrument from the first robotic manipulator and enable an instrument exchange.
2 . The robotic surgical system of claim 1 , wherein:
the user interface is a touch screen,
the gesture detection sensor is a touch sensor,
the presented information is a graphical representation of the surgical instrument, and
the one or more memories have stored thereon, further instructions which, when executed by the one or more processors, cause the one or more processors to determine whether or not a gesture has been detected on the touch screen over the graphical representation of the surgical instrument presented thereon, and in response to a determination that a gesture has been detected, provide the one or more commands to actuate the one or more robotic manipulators in the predetermined manner corresponding to the detected gesture.
3 . The robotic surgical system of claim 1 , wherein:
the user interface is a touch screen,
the gesture detection sensor is a touch sensor,
the presented information is a graphical representation of the one or more robotic manipulators, and
the one or more memories have stored thereon, further instructions which, when executed by the one or more processors, cause the one or more processors to determine whether or not a gesture has been detected on the touch screen over the graphical representation of the robotic manipulator presented thereon, and in response to a determination that a gesture has been detected, provide the one or more commands to actuate the one or more robotic manipulators in the predetermined manner corresponding to the detected gesture.
4 . The robotic surgical system of claim 1 , wherein:
the user interface is a display,
the gesture detection sensor is a camera sensor,
the presented information is a graphical representation of the surgical instrument, and
the one or more memories have stored thereon, further instructions which, when executed by the one or more processors, cause the one or more processors to determine whether or not a gesture has been detected by the camera sensor, and in response to a determination that a gesture has been detected, provide the one or more commands to actuate the one or more robotic manipulators in the predetermined manner corresponding to the detected gesture.
5 . The robotic surgical system of claim 1 , wherein:
the gesture detection sensor is an electric field-based sensor disposed on at least one of the surgical instrument or the one or more robotic manipulators, the gesture detection sensor configured to generate an electric field, and
the one or more memories have stored thereon, further instructions which, when executed by the one or more processors, cause the one or more processors to determine whether or not a gesture has been detected by the electric field-based sensor by receiving a transmission of a change in the electric field generated by the electric field-based sensor, identify a gesture based on the change in the electric field, and provide the one or more commands to actuate the one or more robotic manipulators in a predetermined manner corresponding to the identified gesture.
6 . The robotic surgical system of claim 1 , wherein:
the gesture detection sensor is a radar interaction sensor configured to transmit electromagnetic waves having a predetermined frequency, and
the one or more memories have stored thereon, further instructions which, when executed by the one or more processors, cause the one or more processors to determine whether or not a gesture has been detected by the radar interaction sensor by receiving a transmission of a change in one or more of an amplitude or a signal of the transmitted electromagnetic waves generated by the radar interaction sensor, identify a gesture based on the change in the one or more of an amplitude or a signal of the transmitted electromagnetic waves, and provide the one or more commands to actuate the one or more robotic manipulators in a predetermined manner corresponding to the identified gesture.
7 . The robotic surgical system of claim 1 , further comprising:
a user image capture device coupled to the controller and configured to capture images of the user,
wherein the one or more memories have stored thereon, further instructions which, when executed by the one or more processors, cause the one or more processors to receive the captured images of the user, track one or both of eyes of the user, identify a selection based on the tracked one or both of the eyes of the user, and provide the one or more commands to actuate the one or more robotic manipulators in a predetermined manner corresponding to the identified selection.
8 . The robotic surgical system of claim 1 , wherein the position data further includes a set of coordinates, velocities, location identifiers, and directional identifiers of the first position and the second position in space relative to a fixed location on the map.