Adaptive visualization of contextual targets in surgical video

1 . A computer-implemented method comprising:

extracting, by a system comprising a machine-learning processing system and a procedural control system, as a prediction of a first machine-learning model of the machine-learning processing system or as a user input received by the procedural control system, a proposed region of interest in an image from a video of a surgical procedure based on one or more contextual targets, wherein the procedural control system is configured to receive the user input as a drawing input from a connected device to identify the proposed region of interest as an alternate input source, wherein the first machine-learning model comprises a surgical phase and structure network configured to determine a phase of the surgical procedure in the image, and the one or more contextual targets are determined based on one or more outputs of the surgical phase and structure network;

overlaying a contour around the proposed region of interest as an outline without modifying the image within the proposed region of interest;

modifying the proposed region of interest based on a change to an input that generates the contour;

synthesizing an image adjustment, by a second machine-learning model, based on the proposed region of interest and the image after modifying the proposed region of interest;

generating a modified visualization of the surgical procedure by incorporating the image adjustment in a real-time output of the video of the surgical procedure; and

displaying the video of the surgical procedure with the modified visualization.

2 . The computer-implemented method of claim 1 , wherein the first machine learning model uses weak labels, and the second machine learning model uses weak labels and joint detection and segmentation.

3 . The computer-implemented method of claim 1 , further comprising:

determining motion based on temporal data associated with the image; and

determining a current area of focus as at least a portion of the proposed region of interest based on the motion.

4 . The computer-implemented method of claim 1 , further comprising:

using a depth map to refine the proposed region of interest.

5 . The computer-implemented method of claim 1 , wherein the first machine-learning model is trained based on a training dataset of a plurality of temporally aligned annotated data streams comprising temporal annotations, spatial annotations, and sensor annotations.

6 . The computer-implemented method of claim 1 , further comprising:

performing feature fusion to combine one or more task-specific features of the surgical procedure with one or more temporally aligned features spanning two or more frames.

7 . The computer-implemented method of claim 1 , wherein the user input comprises a drawing input received from one or more devices.

8 . The computer-implemented method of claim 1 , further comprising:

performing eye tracking of a surgeon during the surgical procedure; and

predicting the proposed region of interest based at least in part on a detected area of focus from the eye tracking of the surgeon.

9 . A system comprising:

a data collection system configured to capture a video of a surgical procedure;

a model execution system configured to execute one or more machine-learning models to predict a proposed region of interest in an image from the video of the surgical procedure based on one or more contextual targets, wherein the one or more machine-learning models are configured to use a depth map to refine the proposed region of interest by providing the depth map as an input to an encoder to perform feature fusion with a feature space of an encoding of the image prior to a feature decoder that is used to predict the proposed region of interest; and

an output generator configured to generate a modified visualization of the surgical procedure in a real-time output of the video of the surgical procedure based on the proposed region of interest, wherein the modified visualization comprises an adjustment to one or more enhancement options of: a brightness, a contrast, sharpness, and a size ratio within the proposed region of interest relative to one or more background structures, and the one or more enhancement options are user selectable through a user input.

10 . The system of claim 9 , wherein the system is further configured to determine motion based on temporal data and determine a current area of focus as at least a portion of the proposed region of interest based on the motion.

11 . The system of claim 9 , wherein the one or more machine-learning models are configured to perform feature fusion to combine one or more task-specific features of the surgical procedure with one or more temporally aligned features spanning two or more frames of the video.

12 . The system of claim 9 , further comprising a display configured to output the modified visualization comprising an image adjustment within the proposed region of interest.

13 . A computer program product comprising a non-transitory memory device having computer executable instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform a method comprising:

identifying a proposed region of interest in an image from a video of a surgical procedure;

synthesizing an image adjustment, by one or more machine-learning models, based on the proposed region of interest and the image, wherein the image adjustment is concentrated with a greater intensity near a centroid of the proposed region of interest to appear as a virtual light source, and wherein the one or more machine-learning models are configured to use a depth map to refine the proposed region of interest, the depth map comprises an estimate of three-dimensional depth based on one or more two-dimensional images, and the depth map is provided as an input to an encoder to perform feature fusion with a feature space of an encoding of the image prior to a feature decoder that is used to identify the proposed region of interest; and

generating a modified visualization of the surgical procedure by incorporating the image adjustment in an output of the video of the surgical procedure.

14 . A computer program product comprising a non-transitory memory device having computer executable instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform a method comprising:

identifying a proposed region of interest in an image from a video of a surgical procedure based on one or more contextual targets;

determining motion based on temporal data associated with the image and a current area of focus as at least a portion of the proposed region of interest based on the motion, wherein the one or more contextual targets are determined based on one or more outputs of a surgical phase and structure network;

synthesizing an image adjustment, by one or more machine-learning models, based on the proposed region of interest and the image, wherein the image adjustment is concentrated with a greater intensity near a centroid of the proposed region of interest to appear as a virtual light source; and

generating a modified visualization of the surgical procedure by incorporating the image adjustment in an output of the video of the surgical procedure.

15 . The computer program product of claim 13 , wherein the one or more machine-learning models are trained to perform feature fusion to combine one or more task-specific features of the surgical procedure with one or more temporally aligned features spanning two or more frames, and the feature fusion is based on one or more transform-domain fusion algorithms to implement an image fusion neural network.

16 . The computer program product of claim 13 , wherein the proposed region of interest is identified based on a user input received as a drawing input from a connected device.

17 . The computer program product of claim 13 , wherein execution of the computer executable instructions causes the one or more processors to perform eye tracking of a surgeon during the surgical procedure to track a position of gaze relative to a region of an image being observed and predict the proposed region of interest based at least in part on a detected area of focus from the eye tracking of the surgeon.