Deep-learning-based real-time remaining surgery duration (rsd) estimation
In one aspect, the process receives a current frame of the endoscope video at a current time of the live surgical session, wherein the current time is among a sequence of prediction time points for making continuous RSD predictions during the live surgical session. The process next randomly samples additional frames of the endoscope video corresponding to the elapsed portion of the live surgical session. The process then combines the sampled frames and the current frame in the temporal order to obtain a set of N frames. Next, the process feeds the set of N frames into a trained model for the given surgical procedure. The process subsequently outputs a current RSD prediction based on the set of N frames. Other aspects are also described and claimed.
1. A computer-implemented method for predicting in real-time a remaining surgical duration (RSD) of a live surgical session of a surgical procedure based on a real-time endoscope video of the live surgical session, the method comprising:
sampling a set of N frames of the endoscope video corresponding to the elapsed portion of the live surgical session between a) the beginning of the endoscope video corresponding to the beginning of the live surgical session and b) a current frame corresponding to a current time of the live surgical session;
feeding the set of N frames into a trained RSD machine-learning (ML) model for the surgical procedure;
outputting a current RSD prediction from the trained RSD ML model based on the set of N frames, wherein the outputted current RSD prediction is part of a real-time RSD prediction sequence of the live surgical session; and
predicting a delay based on determining a deviation, as the real-time RSD prediction sequence or a real-time RSD prediction curve deviates from a standard or ideal RSD prediction curve.
2. The computer-implemented method of claim 1 , wherein the deviation is a change in the slope of the real-time RSD prediction curve.
3. The computer-implemented method of claim 2 wherein the change in the slope indicates a potential complication or abnormal event has occurred.
4. The computer-implemented method of claim 3 wherein the abnormal event is a complication associated with a difficult anatomy.
5. The computer-implemented method of claim 3 wherein the abnormal event is bleeding or camera view blocking due to fogging or blood coverage.
6. The computer-implemented method of claim 1 wherein the standard or ideal RSD prediction curve is generated by using a gold standard surgeon.
7. The computer-implemented method of claim 1 further comprising
detecting that the real-time RSD prediction sequence or the real-time RSD prediction curve deviates upward and then drops back down to follow the standard or ideal RSD prediction curve, and in response indicating an abnormal event has occurred.
8. The computer-implemented method of claim 7 , wherein the abnormal event is a surgeon switching an order of the surgical procedure.
9. An article of manufacture comprising a machine-readable medium having stored instructions that configure a processor to predict in real-time a remaining surgical duration (RSD) of a live surgical session of a surgical procedure based on a real-time endoscope video of the live surgical session, by:
sampling a set of N frames of the endoscope video corresponding to the elapsed portion of the live surgical session between a) the beginning of the endoscope video corresponding to the beginning of the live surgical session and b) a current frame corresponding to a current time of the live surgical session;
feeding the set of N frames into a trained RSD machine-learning (ML) model for the surgical procedure;
outputting a current RSD prediction from the trained RSD ML model based on the set of N frames, wherein the outputted current RSD prediction is part of a real-time RSD prediction sequence of the live surgical session; and
predicting a delay based on determining a deviation, as the real-time RSD prediction sequence or a real-time RSD prediction curve deviates from a standard or ideal RSD prediction curve.
10. The article of manufacture of claim 9 wherein the deviation is a change in the slope of the real-time RSD prediction curve.
11. The article of manufacture of claim 10 wherein the change in the slope indicates a potential complication or abnormal event has occurred.
12. The article of manufacture of claim 11 wherein the abnormal event is a complication associated with a difficult anatomy.
13. The article of manufacture of claim 11 wherein the abnormal event is bleeding or camera view blocking due to fogging or blood coverage.
14. The article of manufacture of claim 9 wherein the standard or ideal RSD prediction curve is generated by using a gold standard surgeon.
15. The article of manufacture of claim 9 wherein the machine-readable medium has stored therein further instructions that configure the processor to
detect that the real-time RSD prediction sequence or the real-time RSD prediction curve deviates upward and then drops back down to follow the standard or ideal RSD prediction curve, and in response indicate an abnormal event has occurred.
16. The article of manufacture of claim 15 , wherein the abnormal event is a surgeon switching an order of the surgical procedure.
17. A surgical robotic system comprising:
a processor; and
memory that stores instructions which configure the processor to compute in real-time a remaining surgical duration (RSD) of a live surgical session of a surgical procedure based on a real-time endoscope video of the live surgical session, by:
sampling a set of N frames of the endoscope video corresponding to the elapsed portion of the live surgical session between a) the beginning of the endoscope video corresponding to the beginning of the live surgical session and b) a current frame corresponding to a current time of the live surgical session;
feeding the set of N frames into a trained RSD machine-learning (ML) model for the surgical procedure;
outputting a current RSD prediction from the trained RSD ML model based on the set of N frames, wherein the outputted current RSD prediction is part of a real-time RSD prediction sequence of the live surgical session; and
predicting a delay based on determining a deviation, as the real-time RSD prediction sequence or a real-time RSD prediction curve deviates from a standard or ideal RSD prediction curve.
18. The system of claim 17 wherein the deviation is a change in the slope of the real-time RSD prediction curve.
19. The system of claim 18 wherein the change in the slope indicates a potential complication or abnormal event has occurred.
20. The system of claim 19 wherein the abnormal event is a complication associated with a difficult anatomy, bleeding, or camera view blocking.