Using multiple modalities of surgical data for comprehensive data analytics of a surgical procedure
This patent disclosure provides various embodiments of combining multiple modalities of non-text surgical data in forms of videos, images, and audios in a meaningful manner so that the combined data can be used to perform comprehensive data analytics for a surgical procedure. In some embodiments, the disclosed system can begin by receiving two or more modalities of surgical data during the surgical procedure. The system then time-synchronizes the two or more modalities of surgical data to generate two or more modalities of time-synchronized surgical data. Next, the system converts each modality of the time-synchronized surgical data into a corresponding array of values of a common format. The system then combines the two or more arrays of values to generate a combined set of values. The system subsequently performs comprehensive data analytics on the combined set of values to generate a surgical decision for the surgical procedure.
1 . A computer-implemented method for performing comprehensive data analytics for a surgical procedure, the computer-implemented method comprising:
time-synchronizing multiple modalities of surgical data during a surgical procedure, the multiple modalities of surgical data including a first modality of surgical data and a second modality of surgical data;
converting, by a first segmentation engine, the first modality of surgical data into a first set of alphanumeric values during the surgical procedure, the first modality of surgical data including an audio feed being captured during the surgical procedure;
converting, by a second segmentation engine, the second modality of surgical data into a second set of alphanumeric values during the surgical procedure, the second modality of surgical data including a video feed being captured during the surgical procedure;
combining the first set of alphanumeric values and the second set of alphanumeric values to generate a prediction during the surgical procedure, wherein generating the prediction includes translating an identifiable location of an event detected in the audio feed to a corresponding location of the event within the video feed; and
based on the prediction, highlighting the corresponding location of the event within the video feed.
2 . The computer-implemented method of claim 1 , wherein the audio feed includes a surgeon narrating or discussing the surgical procedure in real-time.
3 . The computer-implemented method of claim 1 , wherein generating the prediction further includes determining a size of an anomaly, an occurrence of a smoking event, or an occurrence of a bleeding event.
4 . The computer-implemented method of claim 1 , wherein the video feed includes real-time video captured by one or more cameras inside a patient's body or near a surgical site, the one or more cameras including an endoscope camera.
5 . The computer-implemented method of claim 1 , further comprising:
time-synchronizing the multiple modalities of surgical data during the surgical procedure, the multiple modalities of surgical data including a third modality of surgical data;
converting, by a third segmentation engine, the third modality of surgical data into a third set of alphanumeric values during the surgical procedure; and
combining the first set of alphanumeric values, the second set of alphanumeric values, and the third set of alphanumeric values to generate the prediction during the surgical procedure.
6 . The computer-implemented method of claim 5 , wherein the third modality is image data associated with the surgical procedure, the image data being generated from pre-op imaging, intra-op real-time imaging, and post-op imaging.
7 . The computer-implemented method of claim 5 further comprising:
combining text data, the first set of alphanumeric values, the second set of alphanumeric values, and the third set of alphanumeric values to generate the prediction related to the event during the surgical procedure, wherein the text data includes patient vitals, patient information, surgeon information, hospital statistics, treatment plans, medication data, test results and progress notes.
8 . A non-transitory computer-readable medium to store instructions that, when executed by a processor of a computer system, cause the computer system to:
time-synchronize multiple modalities of surgical data during a surgical procedure, the multiple modalities of surgical data including a first modality of surgical data and a second modality of surgical data;
convert, by a first segmentation engine, the first modality of surgical data into a first set of alphanumeric values during the surgical procedure, the first modality of surgical data including an audio feed being captured during the surgical procedure;
convert, by a second segmentation engine, the second modality of surgical data into a second set of alphanumeric values during the surgical procedure, the second modality of surgical data including a video feed being captured during the surgical procedure;
combine the first set of alphanumeric values and the second set of alphanumeric values to generate a prediction during the surgical procedure, wherein generating the prediction includes translating an identifiable location of an event detected in the audio feed to a corresponding location of the event within the video feed; and
based on the prediction, highlight the corresponding location of the event within the video feed.
9 . The non-transitory computer-readable medium of claim 8 , wherein, the audio feed includes a surgeon narrating or discussing the surgical procedure in real-time.
10 . The non-transitory computer-readable medium of claim 8 , wherein generating the prediction further includes determining a size of an anomaly, an occurrence of a smoking event, or an occurrence of a bleeding event.
11 . The non-transitory computer-readable medium of claim 8 , wherein the video feed includes real-time video captured by one or more cameras inside a patient's body or near a surgical site, the one or more cameras including an endoscope camera.
12 . The non-transitory computer-readable medium of claim 8 , further comprising instructions that, when executed by the processor of the computer system, cause the computer system to:
time-synchronize the multiple modalities of surgical data during the surgical procedure, the multiple modalities of surgical data including a third modality of surgical data;
convert, by a third segmentation engine, the third modality of surgical data into a third set of alphanumeric values during the surgical procedure; and
combine the first set of alphanumeric values, the second set of alphanumeric values, and the third set of alphanumeric values to generate the prediction during the surgical procedure.
13 . The non-transitory computer-readable medium of claim 12 , wherein the third modality of is image data associated with a surgical procedure, the image data being generated from pre-op imaging, intra-op real-time imaging, and post-op imaging.
14 . The non-transitory computer-readable medium of claim 12 further comprising instructions that, when executed by the processor of the computer system, cause the computer system to:
combine text data, the first set of alphanumeric values, the second set of alphanumeric values, and the third set of alphanumeric values to generate the prediction related to the event during the surgical procedure, wherein the text data includes patient vitals, patient information, surgeon information, hospital statistics, treatment plans, medication data, test results and progress notes.
15 . A system for performing comprehensive data analytics for a surgical procedure, the system comprising:
one or more processors; and
a memory coupled to the one or more processors, wherein the memory stores a set of instructions that, when executed by the one or more processors, cause the system to:
time-synchronize multiple modalities of surgical data during a surgical procedure, the multiple modalities of surgical data including a first modality of surgical data and a second modality of surgical data;
convert, by a first segmentation engine, the first modality of surgical data into a first set of alphanumeric values during the surgical procedure, the first modality of surgical data including an audio feed being captured during the surgical procedure;
convert, by a second segmentation engine, the second modality of surgical data into a second set of alphanumeric values during a surgical procedure, the second modality of surgical data including a video feed being captured during the surgical procedure;
combine the first set of alphanumeric values and the second set of alphanumeric values to generate a prediction during the surgical procedure, wherein generating the prediction includes translating an identifiable location of an event detected in the audio feed to a corresponding location of the event within the video feed; and
based on the prediction, highlight the corresponding location of the event within the video feed.
16 . The system of claim 15 , wherein the audio feed includes a surgeon narrating or discussing the surgical procedure in real-time.
17 . The system of claim 15 , wherein generating the prediction further includes determining a size of an anomaly, an occurrence of a smoking event, or an occurrence of a bleeding event.
18 . The system of claim 15 , wherein the video feed includes real-time video captured by one or more cameras inside a patient's body or near a surgical site, the one or more cameras including an endoscope camera.
19 . The system of claim 15 , further comprising instructions that, when executed by the one or more processors, cause the system to:
time-synchronize the multiple modalities of surgical data during the surgical procedure, the multiple modalities of surgical data including a third modality of surgical data;
convert, by a third segmentation engine, the third modality of surgical data into a third set of alphanumeric values during a surgical procedure; and
combine the first set of alphanumeric values, the second set of alphanumeric values, and the third set of alphanumeric values to generate the prediction during the surgical procedure.
20 . The system of claim 19 , further comprising instructions that, when executed by the one or more processors, cause the system to:
combine text data, the first set of alphanumeric values, the second set of alphanumeric values, and the third set of alphanumeric values to generate the prediction related to the event during the surgical procedure, wherein the third modality of is image data associated with the surgical procedure, the image data being generated from pre-op imaging, intra-op real-time imaging, and post-op imaging, and text data includes patient vitals, patient information, surgeon information, hospital statistics, treatment plans, medication data, test results and progress notes.