Generating augmented visualizations of surgical sites using semantic surgical representations
A surgical action to be performed during a surgical procedure is predicted using machine learning based on images and surgical instrumentation data. An image/video capture device such as an endoscope, a wearable camera, a stationary camera, etc., can be used to capture the image(s). A surgeon can be provided an augmented visualization of the surgical procedure by displaying one or more graphical overlays based on the findings of the machine learning to enhance the surgeon's information.
1 . A computer-implemented method comprising:
predicting, by a first machine learning model, a phase of a surgical procedure being performed based on an input window comprising one or more images from a video of the surgical procedure and instrument data of one or more surgical instruments being used to perform the surgical procedure;
detecting, by a second machine learning model, one or more segmentation masks in the one or more images as separate parts of one or more anatomical structures and the one or more surgical instruments, wherein the second machine learning model comprises an encoder that generates a feature space from the input window and a decoder that outputs the one or more segmentation masks based on the feature space and temporal fusion with temporal information comprising the phase predicted by the first machine learning model;
identifying, by a third machine learning model, an anatomical structure of the one or more anatomical structures and a surgical instrument of the one or more surgical instruments based on the one or more segmentation masks;
predicting, based on the phase, the anatomical structure, and the surgical instrument, a surgical action to be performed as part of the surgical procedure; and
generating a visualization by displaying, the video of the surgical procedure, and a first graphical overlay at a location overlaying the anatomical structure, a second graphical overlay at a location overlaying the surgical instrument, and a trajectory to move the surgical instrument to perform the surgical action that is predicted.
2 . The computer-implemented method of claim 1 , wherein the third machine learning model is trained to predict a location of the anatomical structure associated with a particular type of the surgical procedure.
3 . The computer-implemented method of claim 1 , wherein a confidence score associated with predicting the anatomical structure is determined, and a visual characteristic of the first graphical overlay is modulated based on the confidence score.
4 . The computer-implemented method of claim 1 , wherein the third machine learning model is trained to predict a location of the surgical instrument associated with a particular type of the surgical procedure.
5 . The computer-implemented method of claim 1 , wherein a confidence score associated with predicting the surgical instrument is determined, and a visual characteristic of the second graphical overlay is modulated based on the confidence score.
6 . The computer-implemented method of claim 1 , wherein determining a location of the surgical instrument and a location of the anatomical structure comprises predicting a depth map of a field of view of the video by a depth estimation model that determines a depth estimate of each pixel in the field of view.
7 . The computer-implemented method of claim 1 , further comprising, adjusting an energy level supplied to the surgical instrument based on the surgical action that is predicted.
8 . A system comprising:
a machine learning system configured to use a training dataset to train one or more models to:
predict a phase of a surgical procedure;
segment image data as separate parts of one or more anatomical structures and one or more surgical instruments; and
detect a surgical instrument of the one or more surgical instruments and an anatomical structure of the of the one or more anatomical structures in segmented image data based on the predicted phase;
a data collection system configured to capture a video of a surgical procedure;
a model execution system configured to execute the one or more machine learning models that are trained to:
detect the phase of the surgical procedure that is captured in the video;
detect a location of the surgical instrument and a location of the anatomical structure in a frame of the video by segmenting the frame using a probabilistic heatmap;
predict a surgical action that is to be performed next in the surgical procedure;
an output generator configured to:
provide an indication of the surgical action that is predicted.
9 . The system of claim 8 , wherein the output generator further generates an audio feedback for the surgical action that is predicted.
10 . The system of claim 8 , wherein the output generator is configured to generate a visualization of the surgical procedure by displaying a first graphical overlay at a location overlaying the anatomical structure, a second graphical overlay at a location overlaying the surgical instrument, and a third graphical overlay indicating the surgical action that is predicted.
11 . The system of claim 10 , wherein the visualization comprises a path along which the surgical instrument is to be moved to perform the surgical action that is predicted.
12 . The system of claim 11 , wherein, the path is computed based on one or more tissue attributes of the anatomical structure.
13 . The system of claim 12 , wherein the one or more tissue attributes are determined based on a depth map of a field of view of the video by a depth estimation model that determines a depth estimate of each pixel in the field of view.
14 . The system of claim 8 , wherein the output generator is configured to change at least one from a plurality of attributes of the indication of the surgical action that is predicted, wherein the plurality of attributes comprises, a color, a border-type, a transparency, a priority, and an audible sound.
15 . A computer program product comprising a non-transitory memory device having computer executable instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform a method for prediction of features in surgical data using machine learning, the method comprising:
predicting a phase of a surgical procedure being performed based on an input window comprising one or more images from a video of the surgical procedure and instrument data of one or more surgical instruments being used to perform the surgical procedure;
detecting one or more segmentation masks in the one or more images as separate parts of one or more anatomical structures and the one or more surgical instruments, wherein the machine learning comprises an encoder that generates a feature space from the input window and a decoder that outputs the one or more segmentation masks based on the feature space and temporal fusion with temporal information comprising the phase as predicted;
identifying an anatomical structure of the one or more anatomical structures and a surgical instrument of the one or more surgical instruments based on the one or more segmentation masks;
predicting, based on the phase, the anatomical structure, and the surgical instrument, a surgical action to be performed as part of the surgical procedure; and
generating a visualization to display the video of the surgical procedure with a first graphical overlay at a location overlaying the anatomical structure, a second graphical overlay at a location overlaying the surgical instrument, and a trajectory to move the surgical instrument to perform the surgical action that is predicted.
16 . The computer program product of claim 15 , wherein a location of the surgical instrument and a location of the anatomical structure are predicted based on a depth map of a field of view of the video.
17 . The computer program product of claim 15 , wherein the instructions, when executed by the one or more processors, cause the one or more processors to perform adjusting an energy level supplied to the surgical instrument based on the surgical action that is predicted.
18 . The computer program product of claim 15 , wherein the trajectory of the surgical instrument is computed based on one or more tissue attributes of the anatomical structure.
19 . The computer program product of claim 18 , wherein the machine learning uses weak labels, and/or joint detection and segmentation.
20 . The computer program product of claim 15 , wherein one or more visual attributes of the first graphical overlay and the second graphical overlay are configurable, the one or more visual attributes comprise, color, transparency, shading pattern, and outline.