Prediction of structures in surgical data using machine learning
A location of an anatomical structure in image(s) from a surgical procedure is predicted using machine learning. An image/video capture device such as an endoscope, a wearable camera, a stationary camera, etc., can be used to capture the image(s). A confidence score of the prediction of the machine learning is determined. A surgeon can be provided an augmented visualization of the surgical procedure by displaying one or more graphical overlays based on the findings of the machine learning to enhance the surgeon-s information.
1 . A computer-implemented method comprising:
predicting, by a first machine learning model, a location of an anatomical structure in an input window comprising one or more images from a video of a surgical procedure, the first machine learning model is trained using surgical training data, wherein the first machine learning model predicts the location of the anatomical structure based on a machine-learning generated prediction of a phase of the surgical procedure being performed in the input window, and phase information of the machine-learning generated prediction is used to refine confidence of anatomical structure prediction of the first machine learning model;
predicting, by a second machine learning model, a location of a surgical instrument in the input window, the second machine learning model is trained using the surgical training data;
generating a visualization of the surgical procedure by displaying a first graphical overlay at the location of the anatomical structure and a second graphical overlay at the location of the surgical instrument in the video of the surgical procedure; and
based on a determination that the location of the surgical instrument is within a predetermined threshold from the location of the anatomical structure, providing a user feedback to indicate a proximity of the surgical instrument and the anatomical structure.
2 . The computer-implemented method of claim 1 , wherein the first machine learning model is trained to predict the location of the anatomical structure associated with a particular type of the surgical procedure.
3 . The computer-implemented method of claim 1 , wherein the second machine learning model is trained to predict the location of the surgical instrument associated with a particular type of the surgical procedure.
4 . The computer-implemented method of claim 1 , wherein the second machine learning model predicts the location of the surgical instrument based on an identification of a phase of the surgical procedure being performed.
5 . The computer-implemented method of claim 1 , wherein determining the proximity of the surgical instrument and the anatomical structure comprises predicting, by a third machine learning model, a depth map of a field of view of the video.
6 . The computer-implemented method of claim 1 , wherein the first machine learning model uses weak labels, and the second machine learning model uses weak labels and joint detection and segmentation.
7 . The computer-implemented method of claim 1 , wherein the user feedback is provided based on the anatomical structure being determined as a critical anatomical structure based on one or more attributes of a patient undergoing the surgical procedure.
8 . The computer-implemented method of claim 1 , wherein one or more visual attributes of the visualization are configured based on a confidence score that is refined using the machine-learning generated prediction of the phase.
9 . A system comprising:
a machine learning training system configured to use a training dataset to train:
a first machine learning model to predict a location of an anatomical structure in images from the training dataset; and
a second machine learning model to predict a location of a surgical instrument in the images from the training dataset;
a data collection system configured to capture a video of a surgical procedure;
a model execution system configured to execute the first machine learning model and the second machine learning model to detect the location of the anatomical structure and the location of the surgical instrument in an input window of the video, wherein the first machine earning model predicts the location of the anatomical structure based on a machine-learning generated prediction of a phase of the surgical procedure being performed in the input window, and phase information of the machine-learning generated prediction is used to refine confidence of anatomical structure prediction of the first machine learning model;
an output generator configured to:
based on a determination that the location of the surgical instrument is within a predetermined threshold from the location of the anatomical structure, providing a user feedback by generating a user feedback to indicate a proximity of the surgical instrument and the anatomical structure.
10 . The system of claim 9 , wherein the first machine learning model is trained to predict the location of the anatomical structure associated with a particular type of the surgical procedure.
11 . The system of claim 9 , wherein,
the machine learning training system is further configured to train a third machine learning model to predict a depth map of the images from the training dataset;
the model execution system is further configured to execute the third machine learning model to predict the depth map of a field of view of the video; and
the depth map is used to determine that the location of the surgical instrument is within the predetermined threshold from the location of the anatomical structure.
12 . The system of claim 9 , wherein the output generator is further configured to:
determine a movement vector based on a change in the location of the anatomical structure in the input window of the video;
predict the location of the anatomical structure in a portion of the video that is captured after the input window based on the movement vector; and
displaying the first graphical overlay in a visualization of the video at the predicted location of the anatomical structure.
13 . The system of claim 9 , wherein the user feedback comprises at least one of a visual, audio, tactile, or haptic feedback.
14 . The system of claim 9 , wherein one or more visual attributes of a visualization of the output generator are configured based on a confidence score that is refined using the machine-learning generated prediction of the phase, and the confidence score is computed as a distance transform from a central axis of the anatomical structure.
15 . A computer program product comprising a memory device having computer executable instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform a method for prediction of features in surgical data using machine learning, the method comprising:
predicting, using an encoder-decoder model, a location of an anatomical structure in an input window comprising one or more images from a video of a surgical procedure, the encoder-decoder model is trained using surgical training data, wherein the encoder-decoder model predicts the location of the anatomical structure based on a prediction of a phase of the surgical procedure being performed;
determining a confidence score of the prediction of the location of the anatomical structure, wherein the prediction of the phase the surgical procedure being performed is used to refine the confidence score of the prediction of the location of the anatomical structure; and
generating a visualization of the surgical procedure by displaying a graphical overlay at the location of the anatomical structure in the video of the surgical procedure, one or more visual attributes of the graphical overlay are configured based on the confidence score.
16 . The computer program product of claim 15 , wherein the encoder-decoder model is a first encoder-decoder model, and wherein the method further comprises:
predicting, by a second encoder-decoder model, a location of a surgical instrument in the input window, the second encoder-decoder model is trained using the surgical training data; and
based on a determination that the location of the surgical instrument is within a predetermined threshold from the location of the anatomical structure, providing a user feedback in the visualization.
17 . The computer program product of claim 16 , wherein the user feedback is provided based on the anatomical structure being determined as a critical anatomical structure for the surgical procedure being performed.
18 . The computer program product of claim 16 , wherein the user feedback is provided by generating, in the visualization of the surgical procedure, a second graphical overlay indicative of a proximity of the surgical instrument and the anatomical structure.
19 . The computer program product of claim 18 , wherein determining the proximity of the surgical instrument and the anatomical structure comprises, predicting, by a machine learning model, a depth map of a field of view of the video.
20 . The computer program product of claim 15 , wherein the graphical overlay is displayed in response to the anatomical structure being determined as a critical anatomical structure based for the surgical procedure, and the one or more visual attributes of the graphical overlay comprise color, transparency, shading pattern, text, and outline.