Co-manipulation surgical system having an instrument centering mode for automatic scope movements
Co-manipulation robotic systems are described herein that may be used for assisting with laparoscopic surgical procedures. The co-manipulation robotic systems allow a surgeon to use commercially-available surgical tools while providing benefits associated with surgical robotics. Advantageously, the surgical tools may be seamlessly coupled to the robot arms using a disposable coupler while the reusable portions of the robot arm remain in a sterile drape. Further, the co-manipulation robotic system may operate in multiple modes to enhance usability and safety, while allowing the surgeon to position the instrument directly with the instrument handle and further maintain the desired position of the instrument using the robot arm.
1. A co-manipulation surgical system to assist with laparoscopic surgery performed using a surgical instrument having a handle, an operating end, and an elongated shaft therebetween, the co-manipulation surgical system comprising:
a robot arm comprising a proximal end, a distal end configured to be removably coupled to the surgical instrument, a plurality of links, and a plurality of joints between the proximal end and the distal end; and
a controller operatively coupled to the robot arm and configured to permit the robot arm to be freely moveable responsive to movement at the handle of the surgical instrument for performing laparoscopic surgery, the controller programmed to:
cause the robot arm to maintain a static position in a passive mode responsive to determining that movement of the robot arm due to movement at the handle of the surgical instrument is less than a predetermined amount for at least a predetermined dwell time period;
identify, when the surgical instrument comprises a laparoscope having a field of view, a target surgical instrument within the field of view of the laparoscope based on image data from the laparoscope; and
cause the robot arm to switch to an instrument centering mode where the robot arm moves the laparoscope to maintain the target surgical instrument within the field of view of the laparoscope.
2. The co-manipulation surgical system of claim 1 , wherein the controller is configured to cause the robot arm to automatically switch to a co-manipulation mode responsive to determining that force applied at the robot arm due to force applied at the handle of the surgical instrument exceeds a predetermined threshold, the controller configured to permit the robot arm to be freely moveable in the co-manipulation mode responsive to movement at the handle of the surgical instrument, while applying an impedance to the robot arm in the co-manipulation mode to account for weight of the surgical instrument and the robot arm.
3. The co-manipulation surgical system of claim 1 , wherein the controller is configured to identify the target surgical instrument within the field of view of the laparoscope by detecting a predefined gestural pattern by the target surgical instrument within the field of view of the laparoscope.
4. The co-manipulation surgical system of claim 1 , wherein the predefined gestural pattern comprises positioning of the target surgical instrument within a center portion of the field of view of the laparoscope and maintaining the position of the target surgical instrument within the center portion for at least a predetermined hold period.
5. The co-manipulation surgical system of claim 1 , wherein the controller is configured to identify the target surgical instrument within the field of view of the laparoscope based on user input identifying the target surgical instrument.
6. The co-manipulation surgical system of claim 1 , wherein the controller is configured to distinguish the target surgical instrument from one or more other surgical instruments within the field of view of the laparoscope.
7. The co-manipulation surgical system of claim 1 , wherein, in the instrument centering mode, the controller causes the robot arm to move the laparoscope to maintain the target surgical instrument within a predefined boundary region within the field of view of the laparoscope, such that the robot arm does not move the laparoscope unless the target surgical instrument moves outside of the predefined boundary region.
8. The co-manipulation surgical system of claim 1 , wherein, in the instrument centering mode, the controller causes the robot arm to move the laparoscope by executing a trajectory generation algorithm to generate a trajectory from a current position of the laparoscope to a desired position of the laparoscope, and causing the robot arm to move the laparoscope along the trajectory to maintain the target surgical instrument within the field of view of the laparoscope.
9. The co-manipulation surgical system of claim 8 , wherein the controller is configured to:
permit the robot arm to be freely moveable in a co-manipulation mode responsive to determining that force applied at the robot arm due to force applied at the laparoscope exceeds a predetermined threshold, while applying an impedance to the robot arm in the co-manipulation mode to account for weight of the laparoscope and the robot arm;
record a trajectory of the freely moving robot arm when the movement of the robot arm deviates from the generated trajectory; and
update the trajectory generation algorithm based the recorded trajectory.
10. The co-manipulation surgical system of claim 8 , wherein the generated trajectory comprises moving the robot arm along a longitudinal axis of the laparoscope to maintain the target surgical instrument within the field of view of the laparoscope and within a predetermined resolution threshold.
11. The co-manipulation surgical system of claim 8 , wherein the generated trajectory comprises moving the robot arm along at least one of a longitudinal axis of the laparoscope or an axis perpendicular to the longitudinal axis of the laparoscope to maintain the target surgical instrument within the field of view of the laparoscope.
12. The co-manipulation surgical system of claim 8 , wherein the trajectory is generated by:
measuring a current position of the distal end of the robot arm;
determining a point of entry of the laparoscope into the patient; and
calculating a distance required to move the distal end of the robot arm from its current position to a second position that causes a distal end of the laparoscope to move from its current position to the desired position based on the point of entry and a known length between the distal end of the robot arm and the distal end of the laparoscope.
13. The co-manipulation surgical system of claim 12 , wherein the controller causes the robot arm to move the laparoscope along the trajectory by:
calculating a force required to move the distal end of the robot arm the distance from its current position to the second position; and
applying torque to the at least some joints of the plurality of joints of the robot arm based on the calculated force to move the distal end of the robot arm the distance from its current position to the second position to thereby move the distal end of the laparoscope from its current position to the desired position.
14. The co-manipulation surgical system of claim 8 , wherein the controller is configured to:
detect an offset angle between a camera head of the laparoscope and the laparoscope; and
calibrate the trajectory to correct the offset angle such that movement of the laparoscope along the calibrated trajectory maintains the target surgical instrument within the field of view of the laparoscope.
15. The co-manipulation surgical system of claim 14 , wherein the controller is configured to detect the offset angle by:
causing the robot arm to move along a predetermined trajectory in a known direction in a robot arm coordinate frame;
measuring an actual movement of a static object within the field of view of the laparoscope responsive to movement of the robot arm along the predetermined trajectory; and
comparing the actual movement of the static object with an expected movement of the static object associated with the predetermined trajectory.
16. The co-manipulation surgical system of claim 1 , wherein the controller is configured to identify the target surgical instrument within the field of view of the laparoscope by executing an object segmentation algorithm on the image data from the laparoscope.
17. The co-manipulation surgical system of claim 1 , wherein the controller is configured to cause the robot arm to switch to the instrument centering mode responsive to user input.
18. The co-manipulation surgical system of claim 1 , wherein the controller is configured to:
determine a phase of the laparoscopic surgery;
estimate the target surgical instrument based on the phase of the laparoscopic surgery; and
identify the target surgical instrument within the field of view of the laparoscope based on the estimation and the image data from the laparoscope.
19. The co-manipulation surgical system of claim 1 , wherein the controller is configured to:
determine a phase of the laparoscopic surgery; and
automatically switch to the instrument centering mode responsive to the phase of the laparoscopic surgery.
20. The co-manipulation surgical system of claim 19 , wherein the controller is configured to:
identify one or more anatomical structures within the field of view of the laparoscope based on image data from the laparoscope;
determine the phase of the laparoscope surgery based on the identified one or more anatomical structures; and
cause the robot arm, in the instrument centering mode, to move the laparoscope to maintain the identified one or more anatomical structures within the field of view of the laparoscope.
21. The co-manipulation surgical system of claim 1 , wherein the controller is configured to:
generate an overlay indicative of the target surgical instrument; and
cause the overlay to be displayed over the image data from the laparoscope via a graphical user interface.
22. The co-manipulation surgical system of claim 1 , wherein the controller is configured to:
cause the robot arm to move the laparoscope in a predetermined trajectory; and
compare an actual trajectory of the image data from the laparoscope during movement along the predetermined trajectory with an expected trajectory of the image data associated with the predetermined trajectory to determine an angle of a distal tip of the laparoscope.
23. The co-manipulation surgical system of claim 22 , wherein the predetermined trajectory comprises a circular pattern in a single plane.
24. The co-manipulation surgical system of claim 1 , wherein the controller is configured to identify the target surgical instrument within the field of view of the laparoscope based on image data from the laparoscope using machine learning algorithms executed at the controller.
25. The co-manipulation surgical system of claim 24 , wherein the machine learning algorithms are trained with a database of annotated image data of associated surgical instruments, and wherein the machine learning algorithms are configured to evaluate pixels of the image data from the laparoscope and indicate if the pixels correspond to the target surgical instrument to identify the target surgical instrument.
26. The co-manipulation surgical system of claim 1 , wherein the controller is configured to identify the target surgical instrument within the field of view of the laparoscope in real time.
27. The co-manipulation surgical system of claim 1 , wherein the controller is configured to cause, in the instrument centering mode, the robot arm to move the laparoscope to track the target surgical instrument that is being manually held by a surgeon.
28. The co-manipulation surgical system of claim 27 , further comprising a second robot arm configured to be removably coupled to the target surgical instrument that is being manually held by the surgeon.
29. A method for assisting with laparoscopic surgery, the method comprising:
providing a robot arm comprising a proximal end, a distal end configured to be removably coupled a laparoscope, a plurality of links, and a plurality of joints between the proximal end and the distal end;
permitting, via a controller operatively coupled to the robot arm, the robot arm to be freely moveable responsive to movement at the handle of the laparoscope for performing laparoscopic surgery;
automatically causing, via the controller, the robot arm to maintain a static position in a passive mode responsive to determining that movement of the robot arm due to movement at the handle of the laparoscope is less than a predetermined amount for at least a predetermined dwell time period;
identifying, via the controller, a target surgical instrument within a field of view of the laparoscope based on image data from the laparoscope;
switching, via the controller, the robot arm to an instrument centering mode; and
automatically causing, via the controller while in the instrument centering mode, the robot arm to move the laparoscope to maintain the target surgical instrument within the field of view of the laparoscope.
30. The method of claim 29 , wherein identifying the target surgical instrument within the field of view of the laparoscope comprises detecting, via the controller, a predefined gestural pattern by the target surgical instrument within the field of view of the laparoscope, the predefined gestural pattern comprising positioning of the target surgical instrument within a center portion of the field of view of the laparoscope and maintaining the position of the target surgical instrument within the center portion for at least a predetermined hold period.