Computer system for detecting a tool location in surgical video frame by generating gaussian point on the tool location and predicting surgical actions
Systems, methods, and instrumentalities are disclosed for computer vision-based surgical workflow recognition using natural language processing (NLP) techniques. Surgical video of surgical procedures may be processed and analyzed, for example, to achieve workflow recognition. Surgical phases may be determined based on the surgical video and segmented to generate an annotated video representation. The annotated video representation of the surgical video may provide information associated with the surgical procedure. For example, the annotated video representation may provide information on surgical phases, surgical events, surgical tool usage, and/or the like.
1 . A system for identifying surgical actions in a surgical video, the system comprising:
one or more processors;
a memory storing (a) instructions and (b) a dataset comprising one or more triplets representing surgical video data, wherein each respective triplet comprises instrument data, verb data, and anatomical target data;
an object detection module that uses the one or more processors to detect a tool location in at least one frame of the surgical video data;
an attention module that uses the one or more processors to generate an attention map centered on the tool location wherein the attention module introduces local attention to the tool location by generating a Gaussian point centered on the tool location;
a feature extraction module that implements a two-dimensional convolutional neural network (2D CNN) to extract features from the dataset based on the attention map; and
a prediction module that uses a multi-stage temporal convolutional network (MS-TCN) to recognize, based on the extracted features, one or more surgical actions.
2 . The system of claim 1 , wherein the prediction module recognizes a surgical action and wherein the recognized surgical action is modeled as an action triplet representing activity of the surgical tool.
3 . The system of claim 1 , wherein the object detection module comprises a YOLOv5 (You Only Look Once) neural network to locate one or more bounding boxes of triplets representing areas of interest.
4 . The system of claim 3 , wherein the attention module generates one or more Gaussian maps with the one or more bounding boxes.
5 . The system of claim 1 , wherein the dataset is a laparoscopic dataset or an open surgery dataset.
6 . A method for identifying surgical actions in a surgical video, the method comprising:
obtaining a dataset of surgical video data;
detecting a tool location of a surgical tool in at least one frame of the surgical video data;
obtaining an attention map overlaid on the tool location;
locating one or more bounding boxes of triplets representing areas of interest using a YOLOv5 (You Only Look Once) neural network;
extracting features from the dataset based on the attention map; and
recognizing, based on the extracted features, one or more surgical actions.
7 . The method of claim 6 , wherein the attention map is a Gaussian attention map.
8 . The method of claim 6 , wherein the dataset of surgical video data comprises one or more triplets representing surgical video data.
9 . The method of claim 8 , wherein each respective triplet comprises instrument data, verb data, and anatomical target data.
10 . The method of claim 6 , wherein the attention map overlaid on the tool location is centered on the tool location.
11 . The method of claim 6 , wherein a recognized surgical action from the one or more surgical actions is modeled as an action triplet representing activity of the surgical tool.
12 . The method of claim 6 , wherein the method further comprises introducing local attention to the tool location by generating a Gaussian point centered on the tool location.
13 . The method of claim 6 , wherein the method further comprises generating one or more Gaussian maps with the one or more bounding boxes.
14 . The method of claim 6 , wherein the one or more surgical actions recognized based on the extracted features is recognized using a multi-stage temporal convolutional network (MS-TCN).
15 . The method of claim 6 , wherein the method further comprises:
recognizing, based on the extracted features and using a multi-stage temporal convolutional network (MS-TCN), one or more tool-tissue interactions.
16 . The method of claim 6 , wherein the dataset is a laparoscopic dataset or an open surgery dataset.
17 . A system for identifying surgical actions in a surgical video, the system comprising:
one or more processors;
a memory storing surgical video data;
an object detection module that uses the one or more processors to detect a tool location in at least one frame of the surgical video data, wherein the object detection module comprises a YOLOv 5 (You Only Look Once) neural network to locate one or more bounding boxes of triplets representing areas of interest in the surgical video data;
a feature extraction module that uses Gaussian attention two-dimensional convolutional neural network to extract features associated with the surgical video data based on the one or more bounding boxes; and
a prediction module that uses a multi-stage temporal convolutional network (MS-TCN) to recognize, based on the extracted features, one or more tool-tissue interactions.
18 . The system of claim 17 , wherein the prediction module recognizes a surgical action and wherein the recognized surgical action is modeled as an action triplet representing activity of a surgical tool.