Machine-learning-based visual-haptic feedback system for robotic surgical platforms

1. A computer-implemented method for generating a real-time haptic feedback based on a surgical video containing interactions between surgical tools and tissues, the method comprising:

receiving a surgical video of a surgical procedure;

processing the surgical video to detect, in real time, a first type of surgical tool-tissue interaction captured in the surgical video caused by applying a force on a tissue using one or more surgical tools;

applying a first machine learning model to one or more video images in the surgical video associated with the detected first type of surgical tool-tissue interaction to predict a strength level of the detected first type of surgical tool-tissue interaction; and

outputting the predicted strength level to a surgeon performing the surgical procedure as a real-time haptic feedback signal.

2. The computer-implemented method of claim 1 , wherein processing the surgical video to detect the first type of surgical tool-tissue interaction includes detecting the one or more surgical tools appearing in the surgical video.

3. The computer-implemented method of claim 1 , wherein prior to applying the first machine learning model to the one or more video images, the method further comprises selecting the first machine learning model from a set of machine learning models based on the detected first type of surgical tool-tissue interaction, wherein each machine learning model in the set of machine learning models is constructed to analyze a given type of surgical tool-tissue interaction in a plurality types of surgical tool-tissue interactions.

4. The computer-implemented method of claim 3 , wherein the first machine learning model is trained to classify the one or more video images associated with the first type of surgical tool-tissue interaction as a first strength level within a set of predetermined strength levels defined for the first type of surgical tool-tissue interaction.

5. The computer-implemented method of claim 4 , wherein classifying the one or more video images as the first strength level within the set of predetermined strength levels includes recognizing a first pattern of visual appearance among a set of predefined patterns of visual appearances associated with the first type of surgical tool-tissue interaction.

6. The computer-implemented method of claim 4 , wherein outputting the predicted strength level to the surgeon performing the surgical procedure includes:

converting the predicted strength level into a physical feedback signal; and

communicating the physical feedback signal to the surgeon performing the surgical procedure in real time via a user interface device (UID).

7. The computer-implemented method of claim 6 , wherein the physical feedback signal is a mechanical vibration, and wherein communicating the physical feedback signal to the surgeon via the UID includes transmitting the mechanical vibration to a remote controller of the UID held by the surgeon so that the surgeon can directly feel the physical feedback signal.

8. The computer-implemented method of claim 6 , wherein the method further comprises using different frequencies or different intensities of the mechanical vibration to represent different predetermined strength levels of the first type of surgical tool-tissue interaction.

9. The computer-implemented method of claim 4 , wherein the set of predetermined strength levels includes a maximum strength level, and wherein if the predicted strength level is determined to be above the maximum strength level, the method further comprises configuring the haptic feedback signal as a warning signal.

10. The computer-implemented method of claim 1 , wherein the first type of surgical tool-tissue interaction is caused by applying one of:

a compression force on the tissue using the one or more surgical tools;

a tensile force on the tissue using the one or more surgical tools; and

a combination of a compression force and a tensile force on the tissue using the one or more surgical tools.

11. The computer-implemented method of claim 1 , wherein prior to predicting the strength level of the detected first type of surgical tool-tissue interaction, the method further includes:

processing the surgical video to determine if there is a pause in the detected first type of surgical tool-tissue interaction; and

if so, triggering the first machine learning model to predict the strength level of the detected first type of surgical tool-tissue interaction,

otherwise, delaying applying the first machine learning model to predict the strength level of the detected first type of surgical tool-tissue interaction while continuing detecting a pause in the detected first type of surgical tool-tissue interaction.

12. The computer-implemented method of claim 11 ,

wherein the pause in the detected first type of surgical tool-tissue interaction is caused by the surgeon performing the surgical procedure intentionally holding the tissue steadily for a given period of time; and

wherein the haptic feedback signal outputted during the given period of time remains a constant, thereby allowing the surgeon a sufficient amount of time to understand and react to the predicted strength level.

13. The computer-implemented method of claim 1 , wherein the first type of surgical tool-tissue interaction is associated with one of:

tying a surgical knot during a suture operation;

pulling on the tissue during a cautery operation; and

compressing the tissue during a stapling operation.

14. The computer-implemented method of claim 1 , wherein the method further comprises:

generating a pressure sensor measurement using a pressure sensor integrated at a tip of the one or more surgical tools; and

generating the haptic feedback signal by combining the predicted strength level and the pressure sensor measurement.

15. The computer-implemented method of claim 1 , wherein the surgical video includes one or more of:

an endoscope video;

a laparoscope video; and

a robotic surgical video.

16. An apparatus for generating a real-time haptic feedback based on a surgical video containing interactions between surgical tools and tissues, the apparatus comprising:

one or more processors;

a memory coupled to the one or more processors, wherein the memory stores instructions that, when executed by the one or more processors, cause the apparatus to:

receive a surgical video of a surgical procedure;

process the surgical video to detect, in real time, a first type of surgical tool-tissue interaction captured in the surgical video caused by applying a force on a tissue using one or more surgical tools; and

apply a first machine learning model to one or more video images in the surgical video associated with the detected first type of surgical tool-tissue interaction to predict a strength level of the detected first type of surgical tool-tissue interaction; and

an user interface device including a mechanical component and configured to output the predicted strength level to a surgeon performing the surgical procedure as a real-time haptic feedback signal.

17. The apparatus of claim 16 , wherein the user interface device is configured to output the predicted strength level to the surgeon by:

converting the predicted strength level into a physical feedback signal; and

communicating, through the mechanical component, the physical feedback signal to the surgeon performing the surgical procedure through the mechanical component as the real-time haptic feedback signal.

18. The apparatus of claim 16 ,

wherein the apparatus further includes a pressure sensor integrated at a tip of the one or more surgical tools and configured to generate a pressure sensor measurement; and

wherein the user interface device is further configured to generate the haptic feedback signal by combining the predicted strength level and the pressure sensor measurement.

19. A robotic surgical system, comprising:

one or more surgical tools each coupled to a robotic arm;

an endoscope configured to capture endoscopic videos;

one or more processors configured to:

receive a captured endoscopic video of a surgical procedure performed on the robotic surgical system;

process the endoscopic video to detect, in real time, a first type of surgical tool-tissue interaction captured in the endoscopic video caused by applying a force on a tissue using the one or more surgical tools; and

apply a first machine learning model to one or more video images in the endoscopic video associated with the detected first type of surgical tool-tissue interaction to predict a strength level of the detected first type of surgical tool-tissue interaction; and

a user interface device including a mechanical component and configured to output the predicted strength level to a surgeon performing the surgical procedure as a real-time haptic feedback signal.

20. The robotic surgical system of claim 19 , wherein the user interface device is further configured to output the predicted strength level to the surgeon by:

converting the predicted strength level into a physical feedback signal; and

communicating, through the mechanical component, the physical feedback signal to the surgeon performing the surgical procedure as the real-time haptic feedback signal.