Machine-learning-based visual-haptic system for robotic surgical platforms
Embodiments described herein provide various examples of a machine-learning-based visual-haptic system for constructing visual-haptic models for various interactions between surgical tools and tissues. In one aspect, a process for constructing a visual-haptic model is disclosed. This process can begin by receiving a set of training videos. The process then processes each training video in the set of training videos to extract one or more video segments that depict a target tool-tissue interaction from the training video, wherein the target tool-tissue interaction involves exerting a force by one or more surgical tools on a tissue. Next, for each video segment in the set of video segments, the process annotates each video image in the video segment with a set of force levels predefined for the target tool-tissue interaction. The process subsequently trains a machine-learning model using the annotated video images to obtain a trained machine-learning model for the target tool-tissue interaction.
1. A computer-implemented method for constructing visual-haptic models for interactions between surgical tools and tissues, the method comprising:
receiving a set of training videos;
processing each training video in the set of training videos to extract one or more video segments that depict a target tool-tissue interaction from the training video, wherein the target tool-tissue interaction involves exerting a force by one or more surgical tools on a tissue;
for each video segment in the set of extracted video segments from the set of training videos, annotating each video image in the video segment with a set of force levels predefined for the target tool-tissue interaction; and
training a machine-learning model using the annotated video images as training data to obtain a trained machine-learning model for the target tool-tissue interaction,
wherein the trained machine-learning model is applied to real-time video images to automatically classify a tool-tissue interaction into the set of force levels within the real-time video images.
2. The computer-implemented method of claim 1 , wherein the set of training videos can include:
actual surgical videos performed by surgeons; and
artificially-generated procedure videos created as training data for the machine-learning model.
3. The computer-implemented method of claim 1 , wherein extracting a video segment in the training video includes selecting a video segment that depicts an event of the target tool-tissue interaction from the initial contact between the one or more surgical tools and the tissue until a desired amount of force has been applied by the one or more surgical tools on the tissue.
4. The computer-implemented method of claim 1 , wherein annotating a given video image in the video segment with the set of force levels includes mapping a visual appearance of the target tool-tissue interaction depicted in the given video image to a given force level in the set of force levels.
5. The computer-implemented method of claim 4 , wherein the target tool-tissue interaction includes tying a surgical knot onto a tissue with a pair of forceps during a suture operation, and wherein the set of force levels includes a set of tightness levels of the surgical knot.
6. The computer-implemented method of claim 5 , wherein the visual appearance of the target tool-tissue interaction includes one or more of:
a shape of the surgical knot; and
a shape of the tissue.
7. The computer-implemented method of claim 4 , wherein the target tool-tissue interaction includes pulling on a tissue with a grasper tool in preparation for a cautery operation, and wherein the set of force levels includes a set of tension levels applied on the tissue.
8. The computer-implemented method of claim 7 , wherein the visual appearance of the target tool-tissue interaction includes a curvature of an edge of the tissue under the applied tension.
9. The computer-implemented method of claim 4 , wherein the target tool-tissue interaction includes compressing a tissue with a stapler tool during a stapling operation, and wherein the set of force levels includes a set of compression levels applied on the tissue.
10. The computer-implemented method of claim 9 , wherein the visual appearance of the target tool-tissue interaction includes one or more of: a shape of the jaws of the stapler tool compressing the tissue; and a shape of the tissue being compressed.
11. The computer-implemented method of claim 1 , wherein prior to annotating the set of training videos, the method further comprises establishing the set of force levels for the target tool-tissue interaction by:
establishing a set of visual-appearance standards for the target tool-tissue interaction, wherein each visual-appearance standard in the set of visual-appearance standards correlates a given visual appearance of the target tool-tissue interaction to a corresponding force level of the target tool-tissue interaction; and
mapping the set of established visual-appearance standards to a set of predefined force levels indicative of various degrees of the target tool-tissue interaction.
12. The computer-implemented method of claim 11 , wherein establishing a visual-appearance standard in the set of visual-appearance standards includes:
receiving a set of expert opinions from a set of experts, wherein a given expert opinion from a given expert specifies a strength value assigned to the given visual appearance by the given expert; and
establishing the visual-appearance standard as an average value of the set of assigned strength values.
13. The computer-implemented method of claim 1 , wherein the set of force levels includes at least the following force levels:
a low force level;
a moderate force level; and
a high force level.
14. The computer-implemented method of claim 13 , wherein the set of force levels further includes:
a maximum-safe force level representing a safety threshold for the tissue under the applied force; and
at least one additional force level above the maximum-safe force level.
15. The computer-implemented method of claim 13 , wherein the method further comprises:
using the trained machine-learning model to classify a set of video images associated with the target tool-tissue interaction in a surgical video during a post-procedure analytic process; and
generating a skill score for the target tool-tissue interaction based on the set of classifications for the set of video images.
16. An apparatus for constructing visual-haptic feedback models for interactions between surgical tools and tissues, the apparatus comprising:
a storage module configured to receive a set of training videos;
one or more processors; and
a memory coupled to the storage module and the one or more processors, wherein the memory stores instructions that, when executed by the one or more processors, cause the apparatus to:
process each training video in the set of training videos to extract one or more video segments that depict a target tool-tissue interaction from the training video, wherein the target tool-tissue interaction involves exerting a force by one or more surgical tools on a tissue;
for each video segment in the set of extracted video segments from the set of training videos, annotate each video image in the video segment with a set of force levels predefined for the target tool-tissue interaction; and
train a machine-learning model using the annotated video images as training data to obtain a trained machine-learning model for the target tool-tissue interaction,
wherein the trained machine-learning model is applied to real-time video images to automatically classify a tool-tissue interaction into the set of force levels within the real-time video images.
17. The apparatus of claim 16 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to annotate a given video image in the video segment with the set of force levels by mapping a visual appearance of the target tool-tissue interaction depicted in the given video image to a given force level in the set of force levels.
18. The apparatus of claim 16 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to establish the set of force levels for the target tool-tissue interaction by:
establishing a set of visual-appearance standards for the target tool-tissue interaction, wherein each visual-appearance standard in the set of visual-appearance standards correlates a given visual appearance of the target tool-tissue interaction to a corresponding force level of the target tool-tissue interaction; and
mapping the set of established visual-appearance standards to a set of predefined force levels indicative of various degrees of the target tool-tissue interaction.
19. A robotic-surgical system, comprising:
one or more surgical tools each coupled to a respective robotic arm;
an endoscope configured to capture surgical videos; and
a visual-haptic module including one or more processors, wherein the visual-haptic module is configured to:
process a set of training videos to extract a set of video segments that depict a target tool-tissue interaction, wherein the target tool-tissue interaction involves exerting a force by the one or more surgical tools on a tissue;
for each video segment in the set of extracted video segments, annotate each video image in the video segment with a set of force levels predefined for the target tool-tissue interaction; and
train a machine-learning model using the annotated video images as training data to obtain a trained machine-learning model for the target tool-tissue interaction,
wherein the trained machine-learning model is used to automatically classify a tool-tissue interaction into the set of force levels within the captured surgical videos.
20. The robotic-surgical system of claim 19 , wherein the visual-haptic module is configured to annotate a given video image in the video segment with the set of force levels by mapping a visual appearance of the target tool-tissue interaction depicted in the given video image to a given force level in the set of force levels.
21. The robotic-surgical system of claim 19 , wherein the visual-haptic module is configured to establish the set of force levels for the target tool-tissue interaction by:
establishing a set of visual-appearance standards for the target tool-tissue interaction, wherein each visual-appearance standard in the set of visual-appearance standards correlates a given visual appearance of the target tool-tissue interaction to a corresponding force level of the target tool-tissue interaction; and
mapping the set of established visual-appearance standards to a set of predefined force levels indicative of various degrees of the target tool-tissue interaction.