Methods, systems, and computer readable media for generating and providing artificial intelligence assisted surgical guidance
A method for generating and providing artificial intelligence assisted surgical guidance includes analyzing video images from surgical procedure and training a neural network to identify at least one of anatomical objects, surgical objects, and tissue manipulation in video images. The method includes receiving, by the neural network, a live feed of video images from the surgery. The method further includes classifying, by the neural network, at least one of anatomical objects, surgical objects, and tissue manipulation in the live feed of video images. The method further includes outputting, in real time, surgical guidance based on algorithms created using the classified at least on of anatomical objects, surgical objects, and tissue manipulations in the live feed of video images.
1. A system for generating and providing artificial intelligence assisted surgical guidance, the system comprising:
at least one processor;
a neural network implemented by the at least one processor and trained through analysis of video images from surgical procedures to identify at least one of anatomical objects, surgical objects, and tissue manipulations in the video images;
the neural network being configured to receive, a live feed of video images from a surgery and classify at least one of anatomical objects, surgical objects, and tissue manipulations in the live feed of video images, wherein the neural network includes a backbone comprising a mask-recurrent convolutional neural network (mask RCNN) that receives, as input, a target frame from the live feed of video images and generates as output, image data including a colorization of the target frame that defines likely object contours in terms of color data, the neural network further includes a region proposal network (RPN) that receives, as input, the image data output from the backbone and generates, as output, proposed regions of interest for informing object segmentation, and the neural network further includes mask heads that receive the output from the backbone and the RPN and generates, as outputs, labels for the at least one of the anatomical objects, the surgical objects, and the tissue manipulations in the target frame; and
a surgical guidance generator implemented by the at least one processor for outputting, in real time, guidance based on the classified at least one of anatomical objects, surgical objects, and tissue manipulations in the live feed of video images, wherein outputting the guidance includes overlaying the labels for the at least one of the anatomical objects, the surgical objects, and the tissue manipulations on the live feed of video images or onto a surgical field using augmented reality.
2. The system of claim 1 wherein the neural network is trained to identify tissue including bone, muscle, tendons, organs, blood vessels, and nerve roots, as well as abnormal tissues including tumor.
3. The system of claim 1 wherein the neural network is trained to identify changes in contour of specific tissue types and wherein the surgical guidance generator is configured to output a warning when a change in contour for a tissue type identified in the live video feed nears a damage threshold for the tissue type.
4. The system of claim 1 wherein the neural network is trained to identify and track changes over time and across multiple surgical procedures in at least one of anatomical objects, surgical objects, and tissue manipulations.
5. The system of claim 1 wherein the neural network is trained to identify a pointer instrument having a pointer end that when brought in close proximity to an anatomical object in a live feed of video from a surgery triggers the surgical guidance generator to generate output identifying the anatomical object.
6. The system of claim 1 wherein the surgical guidance generator is trained to interact with a surgeon using voice recognition.
7. The system of claim 1 wherein outputting the surgical guidance displaying movement efficiency metrics and intraoperative metrics predicting success of each surgery.
8. The system of claim 1 wherein outputting the surgical guidance includes incorporating information from additional sources including pre-operative imaging, patient-specific risk factors, surgical object cost data, and intraoperative vital signs.
9. The system of claim 1 wherein mask recurrent convolutional neural network is trained by performing semi-supervised training of the mask recurrent neural network to detect patterns of the at least one of anatomical objects, surgical objects and tissue manipulations in the video frames in combination with supervised training of the mask recurrent convolutional neural network using labeled surgical image frames.