Machine-learning-oriented surgical video analysis system
Embodiments described herein provide various examples of a surgical video analysis system for segmenting surgical videos of a given surgical procedure into shorter video segments and labeling/tagging these video segments with multiple categories of machine learning descriptors. In one aspect, a process for processing surgical videos recorded during performed surgeries of a surgical procedure includes the steps of: receiving a diverse set of surgical videos associated with the surgical procedure; receiving a set of predefined phases for the surgical procedure and a set of machine learning descriptors identified for each predefined phase in the set of predefined phases; for each received surgical video, segmenting the surgical video into a set of video segments based on the set of predefined phases and for each segment of the surgical video of a given predefined phase, annotating the video segment with a corresponding set of machine learning descriptors for the given predefined phase.
1 . A computer-implemented method, the method comprising:
receiving a surgical video of a surgical procedure performed by a surgeon as a plurality of video images;
identifying, in a plurality of subsets of video images of the plurality of video images, a surgical task, a surgical tool, and an anatomy;
for each subset of video images,
creating a metric that establishes an associative relationship among at least two of the surgical task, the surgical tool, or the anatomy within the subset of video images, and
annotating the subset of video images based on the metric, the surgical task, the surgical tool, or the anatomy;
computing an evaluation score for the surgeon according to an overall metric that comprises a union of all metrics from the plurality of subsets of video images; and
training a machine learning (ML) classifier using each annotated subset of video images to detect similar surgical tasks, surgical tools, anatomies, or metrics in other surgical videos.
2 . The computer-implemented method of claim 1 , wherein annotating comprises adding text or captions that describe the metric, the surgical task, the surgical tool, or the anatomy in the subset of video images based on 1) user input, 2) output of an annotation operation that is responsive to input based on the metric, the surgical task, the surgical tool, or the anatomy, or 3) a combination thereof.
3 . The computer-implemented method of claim 1 further comprising segmenting the surgical video into a plurality of phase segments corresponding to a set of predefined phases of the surgical procedure, wherein each phase segment comprises a different subset of video images of the plurality of video images and corresponds to a predefined phase of the set of predefined phases that is associated with the metric, the surgical task, the surgical tool, or the anatomy.
4 . The computer-implemented method of claim 1 further comprising producing a cleaned surgical video by identifying protected health information or non-intraoperative portions within one or more video images of the surgical video and removing the one or more video images from the surgical video.
5 . The computer-implemented method of claim 4 , wherein identifying comprises performing automatic object detection upon the cleaned surgical video.
6 . The computer-implemented method of claim 1 , wherein the ML classifier is a first ML classifier, wherein the method further comprises using the annotated subset of video images to train a second ML classifier to detect similar subsets of video images in other surgical videos.
7 . The computer-implemented method of claim 1 , wherein identifying comprises:
receiving a set of one or more ML descriptors; and
using the ML classifier to detect the surgical task, the surgical tool, or the anatomy within the subset of video images that match the set of one or more ML descriptors.
8 . A system comprising:
at least one processor; and
memory having stored instructions which when executed by the at least one processor causes the system to:
receive a surgical video of a surgical procedure performed by a surgeon as a plurality of video images;
identify, in a plurality of subsets of video images of the plurality of video images, a surgical task, a surgical tool, and an anatomy;
for each subset of video images,
creating a metric that establishes an associative relationship among at least two of the surgical task, the surgical tool, or the anatomy within the subset of video images, and
annotate the subset of video images based on the metric, the surgical task, the surgical tool, or the anatomy;
computing an evaluation score for the surgeon according to an overall metric that comprises a union of all metrics from the plurality of subsets of video images; and
train a machine learning (ML) classifier using each annotated subset of video images to detect similar surgical tasks, surgical tools, anatomies, or metrics in other surgical videos.
9 . The system of claim 8 , wherein the instructions to annotate comprises instructions to add text or captions that describe the metric, the surgical task, the surgical tool, or the anatomy in the subset of video images based on 1) user input, 2) output of an annotation operation that is responsive to input based on the metric, the surgical task, the surgical tool, or the anatomy, or 3) a combination thereof.
10 . The system of claim 8 , wherein the memory comprises further instructions to segment the surgical video into a plurality of phase segments corresponding to a set of predefined phases of the surgical procedure, wherein each phase segment is comprises a different subset of video images of the plurality of video images and corresponds to a predefined phase of the set of predefined phases that is associated with the metric, the surgical task, the surgical tool, or the anatomy.
11 . The system of claim 8 , wherein the memory has further instructions to produce a cleaned surgical video by identifying protected health information or non-intraoperative portions within one or more video images of the surgical video and removing the one or more video images from the surgical video.
12 . The system of claim 11 , wherein the instructions to identify comprises instructions to perform automatic object detection upon the cleaned surgical video.
13 . The system of claim 8 , wherein the ML classifier is a first ML classifier, wherein the memory comprises further instructions to use the annotated subset of video images to train a second ML classifier to detect similar subsets of video images in other surgical videos.
14 . The system of claim 8 , wherein instructions to identify comprises instructions to:
receive a set of one or more ML descriptors; and
use the ML classifier to detect the surgical task, the surgical tool, or the anatomy within the subset of video images that match the set of one or more ML descriptors.
15 . A non-transitory machine-readable medium comprising instructions which when executed by at least one processor of a system, causes the system to:
receive a surgical video of a surgical procedure performed by a surgeon as a plurality of video images;
identify, in a plurality of subsets of video images of the plurality of video images, a surgical task, a surgical tool, and an anatomy;
for each subset of video images,
creating a metric that establishes an associative relationship among at least two of the surgical task, the surgical tool, or the anatomy within the subset of video images, and
annotate the subset of video images based on the metric, the surgical task, the surgical tool, or the anatomy;
compute an evaluation score for the surgeon according to an overall metric that comprises a union of all metrics from the plurality of subsets of video images; and
train a machine learning (ML) classifier each annotated subset of video images to detect similar surgical tasks, surgical tools, anatomies, or metrics in other surgical videos.
16 . The non-transitory machine-readable medium of claim 15 , wherein the instructions to annotate comprises instructions to add text or captions that describe the metric, the surgical task, the surgical tool, or the anatomy in the subset of video images based on 1) user input, 2) output of an annotation operation that is responsive to input based on the metric, the surgical task, the surgical tool, or the anatomy, or 3) a combination thereof.
17 . The non-transitory machine-readable medium of claim 15 comprises further instructions to segment the surgical video into a plurality of phase segments corresponding to a set of predefined phases of the surgical procedure, wherein each phase segment comprises a different subset of video images of the plurality of video images and corresponds to a predefined phase of the set of predefined phases that is associated with the metric, the surgical task, the surgical tool, or the anatomy.
18 . The non-transitory machine-readable medium of claim 15 comprises further instructions to produce a cleaned surgical video by identifying protected health information or non-intraoperative portions within one or more video images of the surgical video and removing the one or more video images from the surgical video.
19 . The non-transitory machine-readable medium of claim 18 , wherein the instructions to identify comprises instructions to perform automatic object detection upon the cleaned surgical video.
20 . The non-transitory machine-readable medium of claim 15 , wherein the ML classifier is a first ML classifier, wherein the non-transitory machine-readable medium comprises further instructions to use the annotated subset of video images to train a second ML classifier to detect similar subsets of video images in other surgical videos.