Machine-learning-based visual-haptic system for robotic surgical platforms
Embodiments described herein provide various examples of a machine-learning-based visual-haptic system for constructing visual-haptic models for various interactions between surgical tools and tissues. In one aspect, a process for constructing a visual-haptic model is disclosed. This process can begin by receiving a set of training videos. The process then processes each training video in the set of training videos to extract one or more video segments that depict a target tool-tissue interaction from the training video, wherein the target tool-tissue interaction involves exerting a force by one or more surgical tools on a tissue. Next, for each video segment in the set of video segments, the process annotates each video image in the video segment with a set of force levels predefined for the target tool-tissue interaction. The process subsequently trains a machine-learning model using the annotated video images to obtain a trained machine-learning model for the target tool-tissue interaction.
1. A computer-implemented method for providing real-time haptic feedback to a surgeon performing a robotic surgery or a tele-surgery, the method comprising:
receiving a surgical video of a surgical procedure performed by a surgeon;
processing the surgical video to detect a tool-tissue interaction involving the surgeon applying a force on a tissue using one or more surgical tools;
recognizing a pause in surgical motions initiated by the surgeon; and
during the pause of the surgical motions:
predicting a strength level of the tool-tissue interaction based on a sequence of video images captured during the pause;
converting the strength level into a physical haptic cue; and
communicating the physical haptic cue to the surgeon as real-time haptic feedback.
2. The computer-implemented method of claim 1 , wherein the tool-tissue interaction is associated with a given surgical task, and wherein detecting the tool-tissue interaction includes detecting a beginning of the given surgical task.
3. The computer-implemented method of claim 2 , wherein detecting the beginning of the given surgical task includes detecting the one or more surgical tools entering video frames of the surgical video.
4. The computer-implemented method of claim 1 , wherein recognizing the pause in the surgical motions includes detecting the tissue being held steadily by the one or more surgical tools for at least a predetermined period of time.
5. The computer-implemented method of claim 1 , wherein predicting the strength level of the detected tool-tissue interaction based on the sequence of video images includes applying a deep-learning model to the sequence of video images associated with the detected tool-tissue interaction, wherein the deep-learning model has been trained to classify each video image in the sequence of video images as a given strength level within a set of predetermined strength levels specified for the detected tool-tissue interaction.
6. The computer-implemented method of claim 5 , wherein while predicting the strength level during the pause of the surgical motions, the strength level outputted by the deep-learning model remains a same strength level within the set of predetermined strength levels, therefore the real-time haptic feedback received by the surgeon during the pause is a constant, which gives the surgeon additional time to understand the physical haptic cue and additional time to react to the physical haptic cue.
7. The computer-implemented method of claim 5 , wherein predicting the strength level during the pause of the surgical motions allows the deep-learning model to process multiple consecutive video images collectively to generate the predicted strength level and with additional processing time, thereby increasing an accuracy of the predicted strength level.
8. The computer-implemented method of claim 5 , wherein the deep-learning model is only activated to classify the sequence of video images associated with the detected surgical tool-tissue interaction during the pause of the surgical motions.
9. The computer-implemented method of claim 5 , wherein classifying each video image as a given strength level within the set of predetermined strength levels includes correlating a visual appearance of the tissue under the applied force to a predefined visual appearance associated with the given strength level among a set of predefined visual appearances associated with the set of predetermined strength levels.
10. The computer-implemented method of claim 9 , wherein a given predefined visual appearance among the set of predefined visual appearances includes:
a curvature associated with the tissue under the applied force; and
a thickness of the tissue under the applied force.
11. The computer-implemented method of claim 10 , wherein the set of predetermined strength levels further includes:
a maximum-safe strength level representing a safety threshold for the tissue under the applied force; and
at least one strength level above the maximum-safe strength level.
12. The computer-implemented method of claim 11 , wherein the physical haptic cue is only communicated to the surgeon when the predicted strength level has reached the safety threshold.
13. The computer-implemented method of claim 5 , wherein the set of predetermined strength levels includes at least the following levels:
a low strength level;
a moderate strength level; and
a high strength level.
14. The computer-implemented method of claim 1 , wherein the physical haptic cue is a mechanical vibration, and wherein communicating the physical haptic cue to the surgeon includes transmitting the mechanical vibration to a remote controller held by the surgeon so that the surgeon can directly feel the real-time haptic feedback.
15. The computer-implemented method of claim 1 , wherein the physical haptic cue further includes one of:
a visual signal that is displayed on a monitor for the surgeon to view; and
an auditory signal that is played through a speaker for the surgeon to hear.
16. The computer-implemented method of claim 1 , wherein the pause in the surgical motions is initiated by the surgeon when the surgeon determines that a sufficient amount of force has been applied to the tissue using the one or more surgical tools.
17. The computer-implemented method of claim 1 , wherein the pause in the surgical motions allows the surgeon to cross-validate the physical haptic cue against a visual appearance of the tissue under the applied force by the one or more surgical tools.
18. The computer-implemented method of claim 1 , wherein the real-time haptic feedback during the pause enables the surgeon to adjust the applied force on the tissue in real time toward a desirable strength level when the physical haptic cue indicates that the predicted strength level of the surgical tool-tissue interaction is either above or below the desirable strength level.
19. An apparatus for providing real-time haptic feedback to a surgeon performing a robotic surgery or a tele-surgery, the apparatus comprising:
one or more processors; and
a memory coupled to the one or more processors, wherein the memory stores instructions that, when executed by the one or more processors, cause the apparatus to:
receive a surgical video of a surgical procedure performed by a surgeon;
process the surgical video to detect a tool-tissue interaction involving the surgeon applying a force on a tissue using one or more surgical tools;
recognize a pause in surgical motions initiated by the surgeon; and
during the pause of the surgical motions:
predict a strength level of the detected tool-tissue interaction based on a sequence of video images captured during the pause;
convert the predicted strength level into a physical haptic cue; and
communicate the physical haptic cue to the surgeon as real-time haptic feedback.
20. The apparatus of claim 19 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to recognize the pause in the surgical motions by detecting the tissue being held steadily by the one or more surgical tools for at least a predetermined period of time.