Detecting events during a surgery

1 . A method comprising:

receiving surgical video of a surgery comprising a plurality of video frames;

receiving data from a robotic surgical system corresponding to the surgery, wherein the data from the robotic surgical system comprises kinematic data corresponding to one or more arms of the robotic surgical system;

identifying, by a first trained machine-learning (“ML”) model, an event during a surgical procedure based on the surgical video and the data from the robotic surgical system;

determining, by the first trained ML model, a subset of the plurality of video frames, the subset of the plurality of video frames corresponding to the event;

determining, by a second trained ML model, a characteristic of the event based on the subset of the plurality of video frames and the data from the robotic surgical system;

generating metadata corresponding to event based on the characteristic of the event;

extracting and storing a video clip comprising the subset of the plurality of video frames from the plurality of video frames; and

extracting and storing robotic log data corresponding to the event.

2 . The method of claim 1 , further comprising:

identifying, by the first trained ML model, a plurality of events during the surgical procedure based on the surgical video;

determining, for each event of the plurality of events by the first trained ML model, a respective subset of the plurality of video frames corresponding to the event;

determining, for each event by the second trained ML model, a respective characteristic of the respective event; and

generating, for each event, metadata corresponding to the respective event based on the respective characteristic of the respective event.

3 . The method of claim 1 , wherein the data from the robotic surgical system comprises audio data.

4 . The method of claim 1 , wherein the event comprises a surgical technique, and wherein the characteristic comprises an assessment of surgical skill.

5 . The method of claim 4 , further comprising:

comparing the assessment to a threshold level of surgical skill; and

responsive to the assessment exceeding the threshold level of surgical skill, including in the metadata an indicator of a training example for the surgical skill.

6 . The method of claim 5 , further comprising notifying one or more surgeons of the surgical video and indicating it as a training example for the surgical skill.

7 . The method of claim 4 , further comprising:

comparing the assessment to a threshold level of surgical skill; and

responsive to the assessment not meeting the threshold level of surgical skill, generating metadata indicating further training to be provided to a surgeon performing the surgical skill in the video.

8 . The method of claim 7 , further comprising outputting a notification indicating that the surgeon may be about to commit an error.

9 . The method of claim 7 , further comprising outputting a notification suggesting one or more training examples for the surgical skill.

10 . The method of claim 7 , further comprising:

identifying the surgeon as having low skill for the surgical skill;

determining the surgeon has low skill for multiple surgical skill; and

outputting a notification indicating the surgeon is a low-skill surgeon.

11 . The method of claim 1 , further comprising:

detecting an error, by the second trained ML model, during the surgery; and

outputting a notification indicating the error.

12 . The method of claim 1 , wherein the event comprises an adverse event.

13 . The method of claim 12 , further comprising outputting a notification to one or more members of a surgical team indicating the adverse event.

14 . A system comprising:

a non-transitory computer-readable medium; and

a processor communicatively coupled to the non-transitory computer-readable medium and configured to execute processor executable instructions stored in the non-transitory computer-readable medium to:

receive surgical video of a surgery comprising a plurality of video frames;

receive data from a robotic surgical system corresponding to the surgery, wherein the data from the robotic surgical system comprises kinematic data corresponding to one or more arms of the robotic surgical system;

identify, using a first trained machine-learning (“ML”) model, an event during a surgical procedure based on the surgical video and the data from the robotic surgical system;

determine, using the first trained ML model, a subset of the plurality of video frames, the subset of the plurality of video frames corresponding to the event;

determine, using a second trained ML model, a characteristic of the event based on the subset of the plurality of video frames and the data from the robotic surgical system;

generate metadata corresponding to event based on the characteristic of the event;

extract and store a video clip comprising the subset of the plurality of video frames from the plurality of video frames; and

extract and store robotic log data corresponding to the event.

15 . The system of claim 14 , further comprising:

identifying, by the first trained ML model, a plurality of events during the surgical procedure based on the surgical video;

determining, for each event of the plurality of events by the first trained ML model, a respective subset of the plurality of video frames corresponding to the event;

determining, for each event by the second trained ML model, a respective characteristic of the respective event; and

generating, for each event, metadata corresponding to the respective event based on the respective characteristic of the respective event.

16 . The system of claim 14 , wherein the data from the robotic surgical system comprises audio data.

17 . The system of claim 14 , wherein the event comprises a surgical technique, and wherein the characteristic comprises an assessment of surgical skill.

18 . The system of claim 17 , further comprising:

comparing the assessment to a threshold level of surgical skill; and

responsive to the assessment exceeding the threshold level of surgical skill, including in the metadata an indicator of a training example for the surgical skill.

19 . The system of claim 17 , further comprising:

comparing the assessment to a threshold level of surgical skill; and

responsive to the assessment not meeting the threshold level of surgical skill generating metadata indicating further training to be provided to a surgeon performing the surgical skill in the video.

20 . The system of claim 14 , further comprising:

detecting an error, by the second trained ML model, during the surgery; and

outputting a notification during the surgery indicating the error.

21 . The system of claim 14 , wherein the event comprises a surgical technique, wherein the surgical technique is a non-standard technique.

22 . The system of claim 14 , wherein the event comprises an abnormal anatomical feature.

23 . The system of claim 14 , wherein the event comprises an adverse event.

24 . A non-transitory computer-readable medium comprising processor executable instructions stored in the non-transitory computer-readable medium to:

receive surgical video of a surgery comprising a plurality of video frames;

receive data from a robotic surgical system corresponding to the surgery, wherein the data from the robotic surgical system comprises kinematic data corresponding to one or more arms of the robotic surgical system;

identify, using a first trained machine-learning (“ML”) model, an event during a surgical procedure based on the surgical video and the data from the robotic surgical system;

determine, using the first trained ML model, a subset of the plurality of video frames, the subset of the plurality of video frames corresponding to the event;

determine, using a second trained ML model, a characteristic of the event based on the subset of the plurality of video frames and the data from the robotic surgical system;

generate metadata corresponding to event based on the characteristic of the event;

extract and store a video clip comprising the subset of the plurality of video frames from the plurality of video frames; and

extract and store robotic log data corresponding to the event.