Methods and systems for using multiple data structures to process surgical data
The present disclosure relates to processing data streams from a surgical procedure using multiple interconnected data structures to generate and/or continuously update an electronic output. Each surgical data structure is used to determine a current node associated with a characteristic of a surgical procedure and present relevant metadata associated with the surgical procedure. Each surgical data structure includes at least one node interconnected to one or more nodes of another data structure. The interconnected nodes between one or more data structures includes relational metadata associated with the surgical procedure.
1. A computer-implemented method comprising:
accessing, by a processor, a surgical dataset associated with a surgical procedure, the surgical dataset comprising a surgical video of the surgical procedure being performed using a surgical tool;
creating, by the processor, a surgical map of the surgical procedure by analyzing the surgical video using a trained machine learning algorithm that is trained using training data with pixel-level labels from a segmented procedure video stream, wherein the surgical map comprises at least:
a first data structure that comprises a first plurality of nodes and a first plurality of edges, wherein a first node from the first data structure represents a procedural state in the surgical procedure;
a second data structure that comprises a second plurality of nodes and a second plurality of edges, wherein a second node from the second data structure represents the surgical tool used in the surgical procedure;
adding, by the processor, an interconnecting edge between the first node and the second node based on a surgical event detected in the surgical video, wherein the interconnecting edge is associated with a relational metadata representing a use of the surgical tool of the second node in the procedural state of the first node;
augmenting, by the processor, the surgical video by including an electronic data to a portion of the surgical video comprising a set of pixels, the electronic data describing an operational note based at least on the procedural state of the first node, a data associated with the second node, and the relational metadata associated with the interconnecting edge between the first node and the second node, the portion comprising the use of the surgical tool of the second node in the procedural state of the first node; and
outputting, by the processor, the electronic data during playback of the portion of the surgical video on a display.
2. The computer-implemented method of claim 1 , wherein the surgical dataset further comprises one or more data streams captured during the surgical procedure, each of the one or more data streams obtained from a corresponding data capture device, wherein the one or more data streams comprise:
video streams captured by one or more cameras;
audio streams captured by one or more microphones; and/or
signals from electronic devices comprising medical equipment used during the surgical procedure.
3. The computer-implemented method of claim 2 , wherein the electronic data is associated with a second portion of at least one of the one or more data streams, the second portion corresponding to the use of the surgical tool of the second node in the procedural state of the first node.
4. The computer-implemented method of claim 2 , wherein the one or more data streams further comprise user inputs provided by operating room personnel via a wearable device or non-wearable device located inside or near an operating room while the surgical procedure is performed.
5. The computer-implemented method of claim 1 , wherein the first plurality of edges in the first data structure represents a transition of procedural states during the surgical procedure.
6. The computer-implemented method of claim 1 , wherein the surgical map comprises additional data structures, each additional data structure comprising a plurality of nodes and a plurality of edges.
7. The computer-implemented method of claim 1 , further comprising updating an electronic medical record to include the electronic data.
8. The computer-implemented method of claim 1 , wherein outputting the electronic data during playback of the portion comprises augmenting the surgical video with the electronic data, and either storing an augmented video stream or transmitting the augmented video stream for playback.
9. The computer-implemented method of claim 1 , wherein the trained machine learning algorithm uses separate machine learning models for each data structure from the surgical map.
10. A system comprising:
a multi-dimensional artificial intelligence system;
a surgical operating system configured to interface with one or more cameras;
and one or more processors coupled with the multi-dimensional artificial intelligence system, and the surgical operating system, the one or more processors are configured to perform a method comprising:
accessing a surgical dataset associated with a surgical procedure, the surgical dataset comprising a surgical video of the surgical procedure being performed as captured by the one or more cameras;
creating a surgical map of the surgical procedure by analyzing the surgical video using the multi-dimensional artificial intelligence system that is trained using training data with pixel-level labels from a segmented procedure video stream, wherein the surgical map comprises at least:
a first data structure that comprises a first plurality of nodes and a first plurality of edges, wherein a first node from the first data structure represents a first procedural feature of the surgical procedure; and
a second data structure that comprises a second plurality of nodes and a second plurality of edges, wherein a second node from the second data structure represents second procedural feature of the surgical procedure;
adding an interconnecting edge between the first node and the second node based on a surgical event detected in the surgical video, wherein the interconnecting edge is associated with a relational metadata representing a use of the first procedural feature of the first node in connection with the second procedural feature of the second node;
augmenting the surgical video by including an electronic data to a portion of the surgical video comprising a set of pixels, wherein the electronic data describes an operational note based at least on the first procedural feature, the second procedural feature, and the relational metadata associated with the interconnecting edge between the first node and the second node, the portion comprising the use of the first procedural feature in connection with the second procedural feature; and
outputting the electronic data during playback of the portion of the surgical video on a display.
11. The system of claim 10 , wherein the first data structure and the second data structure represent one from a group of procedural features comprising procedural states, surgical tools, anatomical structures, phonemes, medical personnel, and surgical actions.
12. The system of claim 10 , wherein outputting the electronic data during playback of the portion comprises augmenting the surgical video with the electronic data, and either storing an augmented video stream or transmitting the augmented video stream for playback.
13. The system of claim 10 , wherein the surgical dataset further comprises one or more data streams captured during the surgical procedure, each of the one or more data streams obtained from a corresponding data capture device.
14. The system of claim 13 , wherein the electronic data is associated with a second portion of at least one of the one or more data streams, the second portion corresponding to the use of the second procedural feature of the second node with the first procedural feature of the first node.
15. The system of claim 13 , wherein the one or more processors are further configured to update an electronic medical record to include the operational note.
16. A computer program product comprising a non-transitory computer storage medium having computer executable instructions stored thereon, the computer executable instructions when executed by one or more processors cause the one or more processors to perform a method comprising:
creating a surgical map of a surgical procedure by analyzing a surgical video of the surgical procedure using a trained machine learning algorithm that is trained using training data with pixel-level labels from a segmented procedure video stream, wherein the surgical map comprises at least:
a first data structure that comprises a first plurality of nodes, wherein a first node from the first data structure represents a first procedural feature of the surgical procedure; and
a second data structure that comprises a second plurality of nodes, wherein a second node from the second data structure represents second procedural feature of the surgical procedure;
adding an interconnecting edge between the first node and the second node based on a surgical event detected in the surgical video, wherein the interconnecting edge is associated with a relational metadata representing a use of the first procedural feature of the first node in connection with the second procedural feature of the second node;
augmenting the surgical video by including an electronic data to a portion of the surgical video comprising a set of pixels, the electronic data describing an operational note based at least on the first procedural feature, the second procedural feature, and the relational metadata associated with the interconnecting edge between the first node and the second node, the portion of the surgical video comprising the use of the first procedural feature in connection with the second procedural feature; and
outputting the electronic data during playback of the portion of the surgical video on a display.
17. The computer program product of claim 16 , wherein the surgical video is streamed live.
18. The computer program product of claim 16 , wherein the surgical video is accessed from a storage device.
19. The computer program product of claim 16 , wherein the method further comprises updating an electronic medical record to include the operational note.