Machine-learning-based visual-haptic feedback system for robotic surgical platforms
View Patent ↗Embodiments described herein provide various examples of a visual-haptic feedback system for generating a haptic feedback signal based on captured endoscopy images. In one aspect, the process for generating the haptic feedback signal includes the steps of: receiving an endoscopic video captured for a surgical procedure performed on a robotic surgical system; detecting a surgical task in the endoscopic video involving a given type of surgical tool-tissue interaction; selecting a machine learning model constructed for analyzing the given type of surgical tool-tissue interaction; for a video image associated with the detected surgical task depicting the given type of surgical tool-tissue interaction, applying the selected machine learning model to the video image to predict a strength level of the depicted surgical tool-tissue interaction; and then providing the predicted strength level to a surgeon performing the surgical task as a haptic feedback signal for the given type of surgical tool-tissue interaction.
1. A computer-implemented method for generating a haptic feedback based on a captured endoscopic video depicting an interaction between one or more surgical tools and a tissue during a robotic surgical procedure, the method comprising:
receiving an endoscopic video captured for a surgical procedure performed on a robotic surgical system;
detecting a surgical task in the endoscopic video that involves a given type of surgical tool-tissue interaction caused by applying a force on a tissue using one or more surgical tools;
selecting a machine learning model from a set of machine learning models based on the detected surgical task, wherein each machine learning model in the set of machine learning models is trained to analyze a given type of surgical tool-tissue interaction;
for a video image or a sequence of video images associated with the detected surgical task depicting the given type of surgical tool-tissue interaction, applying the selected machine learning model to the video image or the sequence of video images to predict a strength level of the depicted surgical tool-tissue interaction; and
providing the predicted strength level to a surgeon performing the surgical task as a haptic feedback signal for the given type of surgical tool-tissue interaction.
2. The computer-implemented method of claim 1 , wherein detecting the surgical task in the endoscopic video involving the given type of surgical tool-tissue interaction includes detecting one or more surgical tools coming into view.
3. The computer-implemented method of claim 1 , wherein the selected machine learning model is trained to classify a video image or a sequence of video images depicting the given type of surgical tool-tissue interaction as one of a set of predetermined strength levels defined for the given type of surgical tool-tissue interaction.
4. The computer-implemented method of claim 3 , wherein applying the selected machine learning model to the video image or the sequence of video images to predict the strength level of the depicted surgical tool-tissue interaction includes classifying the video image or the sequence of video images as one of the set of predetermined strength levels for the given type of surgical tool-tissue interaction based on analyzing a visual appearance of the depicted surgical tool-tissue interaction.
5. The computer-implemented method of claim 3 , wherein the set of predetermined strength levels includes a maximum strength level, and wherein if the predicted strength level is determined to be above the maximum strength level, the method further comprises generating a warning signal to be communicated to the surgeon.
6. The computer-implemented method of claim 1 , wherein providing the predicted strength level to the surgeon performing the surgical task includes:
converting the predicted strength level by the selected machine learning model into a physical feedback signal; and
communicating the converted physical feedback signal to the surgeon performing the surgical task via a user interface device (UID) of the robotic surgical system.
7. The computer-implemented method of claim 6 , wherein the physical feedback signal is a mechanical vibration, and wherein communicating the converted physical feedback signal to the surgeon via the UID of the robotic surgical system includes transmitting the mechanical vibration to a remote controller of the UID held by the surgeon so that the surgeon can directly feel the physical feedback signal.
8. The computer-implemented method of claim 7 , wherein the method further comprises using different frequencies or different intensities of the mechanical vibration to represent different predicted strength levels of the given type of surgical tool-tissue interaction.
9. The computer-implemented method of claim 6 , wherein the physical feedback signal is an auditory signal, and wherein communicating the converted physical feedback signal to the surgeon via the UID of the robotic surgical system includes transmitting the auditory signal using a speaker of the UID so that the surgeon can directly hear the physical feedback signal.
10. The computer-implemented method of claim 9 , wherein the method further comprises using different volumes or different pitches of the auditory signal to represent different predicted strength levels of the given type of surgical tool-tissue interaction.
11. The computer-implemented method of claim 1 , wherein the given type of surgical tool-tissue interaction is caused by applying a compression force on the tissue using the one or more surgical tools.
12. The computer-implemented method of claim 1 , wherein the given type of surgical tool-tissue interaction is caused by applying a tensile force on the tissue using the one or more surgical tools.
13. The computer-implemented method of claim 1 , wherein the given type of surgical tool-tissue interaction is caused by applying the force using a single surgical tool on the tissue.
14. The computer-implemented method of claim 1 , wherein the given type of surgical tool-tissue interaction is caused by applying the force using two or more surgical tools on the tissue.
15. The computer-implemented method of claim 1 , wherein the given type of surgical tool-tissue interaction is associated with one of:
tying a surgical knot during a suture operation;
pulling on the tissue during a cautery operation; and
compressing the tissue during a stapling operation.
16. The computer-implemented method of claim 1 , wherein prior to using the selected machine learning model, the method further comprises training the selected machine learning model by:
receiving a set of endoscopic videos;
processing each of the endoscopic videos to extract from the endoscopic video one or more segments that depict the given type of tool-tissue interaction;
for each of the extracted video segments, annotating video images depicting the given type of tool-tissue interaction according to a set of predetermined strength levels; and
using the annotated video images as ground truth information to train a machine learning model to analyze the given type of surgical tool-tissue interaction.
17. A system for generating a haptic feedback based on captured endoscopy images depicting an interaction between one or more surgical tools and a tissue during a robotic surgical procedure, the system comprising:
one or more processors;
a memory coupled to the one or more processors;
a receiving module configured to receive an endoscopic video captured for a surgical procedure performed on a robotic surgical system;
a detection module configured to detect a surgical task in the endoscopic video that involves a given type of surgical tool-tissue interaction caused by applying a force on a tissue using one or more surgical tools;
a selection module configured to select a machine learning model from a set of machine learning models based on the detected surgical task, wherein each machine learning model in the set of machine learning models is trained to analyze a given type of surgical tool-tissue interaction;
a machine learning module configured to, for a video image or a sequence of video images associated with the detected surgical task depicting the given type of surgical tool-tissue interaction, apply the selected machine learning model to the video image or the sequence of video images to predict a strength level of the depicted surgical tool-tissue interaction; and
a user interface module configured to provide the predicted strength level to a surgeon performing the surgical task as a haptic feedback signal for the given type of surgical tool-tissue interaction.
18. The system of claim 17 , wherein the system further comprises a machine-learning-model training module configured to train the selected machine learning model by:
receiving a set of endoscopic videos;
processing each of the endoscopic videos to extract from the endoscopic video one or more segments that depict the given type of tool-tissue interaction;
for each of the extracted video segments, annotating video images depicting the given type of tool-tissue interaction according to a set of predetermined strength levels; and
using the annotated video images as ground truth information to train a machine learning model to analyze the given type of surgical tool-tissue interaction.
19. A robotic surgical system, comprising:
one or more surgical tools each coupled to a robotic arm;
an endoscope configured to capture endoscopic videos;
a receiving module configured to receive a captured endoscopic video for a surgical procedure performed on the robotic surgical system;
a detection module configured to detect a surgical task in the endoscopic video that involves a given type of surgical tool-tissue interaction caused by applying a force on a tissue using the one or more surgical tools;
a selection module configured to select a machine learning model from a set of machine learning models based on the detected surgical task, wherein each machine learning model in the set of machine learning models is trained to analyze a given type of surgical tool-tissue interaction;
a machine learning module configured to, for a video image or a sequence of video images associated with the detected surgical task depicting the given type of surgical tool-tissue interaction, apply the selected machine learning model to the video image or the sequence of video images to predict a strength level of the depicted surgical tool-tissue interaction; and
a user interface module configured to provide the predicted strength level to a surgeon performing the surgical task as a haptic feedback signal for the given type of surgical tool-tissue interaction.
20. The robotic surgical system of claim 19 , wherein the robotic surgical system further comprises a machine-learning-model training module configured to train the selected machine learning model by:
receiving a set of endoscopic videos;
processing each of the endoscopic videos to extract from the endoscopic video one or more segments that depict the given type of tool-tissue interaction;
for each of the extracted video segments, annotating video images depicting the given type of tool-tissue interaction according to a set of predetermined strength levels; and
using the annotated video images as ground truth information to train a machine learning model to analyze the given type of surgical tool-tissue interaction.