Adaptive visualization of contextual targets in surgical video
An aspect includes a computer-implemented method that predicts a proposed region of interest in an image from a video of a surgical procedure based on one or more contextual targets. An image adjustment is synthesized based on the proposed region of interest and the image. A modified visualization of the surgical procedure is generated by incorporating the image adjustment in a real-time output of the video of the surgical procedure. The video of the surgical procedure is displayed with the modified visualization.
1 . A computer-implemented method comprising:
extracting, by a system comprising a machine-learning processing system and a procedural control system, as a prediction of a first machine-learning model of the machine-learning processing system or as a user input received by the procedural control system, a proposed region of interest in an image from a video of a surgical procedure based on one or more contextual targets, wherein the procedural control system is configured to receive the user input as a drawing input from a connected device to identify the proposed region of interest as an alternate input source, wherein the first machine-learning model comprises a surgical phase and structure network configured to determine a phase of the surgical procedure in the image, and the one or more contextual targets are determined based on one or more outputs of the surgical phase and structure network;
overlaying a contour around the proposed region of interest as an outline without modifying the image within the proposed region of interest;
modifying the proposed region of interest based on a change to an input that generates the contour;
synthesizing an image adjustment, by a second machine-learning model, based on the proposed region of interest and the image after modifying the proposed region of interest;
generating a modified visualization of the surgical procedure by incorporating the image adjustment in a real-time output of the video of the surgical procedure; and
displaying the video of the surgical procedure with the modified visualization.
2 . The computer-implemented method of claim 1 , wherein the first machine learning model uses weak labels, and the second machine learning model uses weak labels and joint detection and segmentation.
3 . The computer-implemented method of claim 1 , further comprising:
determining motion based on temporal data associated with the image; and
determining a current area of focus as at least a portion of the proposed region of interest based on the motion.
4 . The computer-implemented method of claim 1 , further comprising:
using a depth map to refine the proposed region of interest.
5 . The computer-implemented method of claim 1 , wherein the first machine-learning model is trained based on a training dataset of a plurality of temporally aligned annotated data streams comprising temporal annotations, spatial annotations, and sensor annotations.
6 . The computer-implemented method of claim 1 , further comprising:
performing feature fusion to combine one or more task-specific features of the surgical procedure with one or more temporally aligned features spanning two or more frames.
7 . The computer-implemented method of claim 1 , wherein the user input comprises a drawing input received from one or more devices.
8 . The computer-implemented method of claim 1 , further comprising:
performing eye tracking of a surgeon during the surgical procedure; and
predicting the proposed region of interest based at least in part on a detected area of focus from the eye tracking of the surgeon.
9 . A system comprising:
a data collection system configured to capture a video of a surgical procedure;
a model execution system configured to execute one or more machine-learning models to predict a proposed region of interest in an image from the video of the surgical procedure based on one or more contextual targets, wherein the one or more machine-learning models are configured to use a depth map to refine the proposed region of interest by providing the depth map as an input to an encoder to perform feature fusion with a feature space of an encoding of the image prior to a feature decoder that is used to predict the proposed region of interest; and
an output generator configured to generate a modified visualization of the surgical procedure in a real-time output of the video of the surgical procedure based on the proposed region of interest, wherein the modified visualization comprises an adjustment to one or more enhancement options of: a brightness, a contrast, sharpness, and a size ratio within the proposed region of interest relative to one or more background structures, and the one or more enhancement options are user selectable through a user input.
10 . The system of claim 9 , wherein the system is further configured to determine motion based on temporal data and determine a current area of focus as at least a portion of the proposed region of interest based on the motion.
11 . The system of claim 9 , wherein the one or more machine-learning models are configured to perform feature fusion to combine one or more task-specific features of the surgical procedure with one or more temporally aligned features spanning two or more frames of the video.
12 . The system of claim 9 , further comprising a display configured to output the modified visualization comprising an image adjustment within the proposed region of interest.
13 . A computer program product comprising a non-transitory memory device having computer executable instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform a method comprising:
identifying a proposed region of interest in an image from a video of a surgical procedure;
synthesizing an image adjustment, by one or more machine-learning models, based on the proposed region of interest and the image, wherein the image adjustment is concentrated with a greater intensity near a centroid of the proposed region of interest to appear as a virtual light source, and wherein the one or more machine-learning models are configured to use a depth map to refine the proposed region of interest, the depth map comprises an estimate of three-dimensional depth based on one or more two-dimensional images, and the depth map is provided as an input to an encoder to perform feature fusion with a feature space of an encoding of the image prior to a feature decoder that is used to identify the proposed region of interest; and
generating a modified visualization of the surgical procedure by incorporating the image adjustment in an output of the video of the surgical procedure.
14 . A computer program product comprising a non-transitory memory device having computer executable instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform a method comprising:
identifying a proposed region of interest in an image from a video of a surgical procedure based on one or more contextual targets;
determining motion based on temporal data associated with the image and a current area of focus as at least a portion of the proposed region of interest based on the motion, wherein the one or more contextual targets are determined based on one or more outputs of a surgical phase and structure network;
synthesizing an image adjustment, by one or more machine-learning models, based on the proposed region of interest and the image, wherein the image adjustment is concentrated with a greater intensity near a centroid of the proposed region of interest to appear as a virtual light source; and
generating a modified visualization of the surgical procedure by incorporating the image adjustment in an output of the video of the surgical procedure.
15 . The computer program product of claim 13 , wherein the one or more machine-learning models are trained to perform feature fusion to combine one or more task-specific features of the surgical procedure with one or more temporally aligned features spanning two or more frames, and the feature fusion is based on one or more transform-domain fusion algorithms to implement an image fusion neural network.
16 . The computer program product of claim 13 , wherein the proposed region of interest is identified based on a user input received as a drawing input from a connected device.
17 . The computer program product of claim 13 , wherein execution of the computer executable instructions causes the one or more processors to perform eye tracking of a surgeon during the surgical procedure to track a position of gaze relative to a region of an image being observed and predict the proposed region of interest based at least in part on a detected area of focus from the eye tracking of the surgeon.