Cascade stage boundary awareness networks for surgical workflow analysis
Techniques are described for improving computer-assisted surgical (CAS) systems, particularly, to recognize surgical phases in a video of a surgical procedure. A CAS system includes cameras that provide video stream of a surgical procedure. According to one or more aspects the surgical phases are automatically detected in the video stream using a machine learning model. Particularly, the machine learning model includes a boundary aware cascade stage network to perform surgical phase recognition.
1 . A system comprising:
a memory device; and
one or more processors coupled with the memory device, the one or more processors configured to:
encode a frame of a video of a surgical procedure into a plurality of features;
provide the features to a boundary supervision branch of a model;
provide the features to a cascade of temporal stages of the model, wherein the cascade of temporal stages comprises a dilated convolution stage in series with one or more reweighted dilated convolution stages; and
perform fusion of an output of the cascade of temporal stages with an output of the boundary supervision branch to predict a surgical phase of the surgical procedure depicted in the frame.
2 . The system of claim 1 , wherein the boundary supervision branch is trained to detect a transition condition between surgical phases.
3 . The system of claim 2 , wherein the boundary supervision branch is trained in parallel with the cascade of temporal stages according to a loss function that changes over a plurality of epochs.
4 . The system of claim 1 , wherein the frame is encoded by an encoder trained using self-supervised learning.
5 . The system of claim 4 , wherein the self-supervised learning comprises a student network that attempts to predict from an augmented image and a target signal generated by a teacher network for a same image under different augmentation, wherein only the student network is updated through backpropagation, while an exponential moving average is used to update the teacher network.
6 . The system of claim 5 , wherein phase labels are used to update the student network and a contribution of phase supervision is reduced as training progresses according to a classification loss function.
7 . The system of claim 1 , wherein the video of the surgical procedure is captured by an endoscopic camera from inside of a patient's body.
8 . The system of claim 1 , wherein the video of the surgical procedure is captured by a camera from outside of a patient's body.
9 . A computer-implemented method comprising:
encoding a frame of a video of a surgical procedure into a plurality of features;
providing the features to a boundary supervision branch of a model;
providing the features to a cascade of temporal stages of the model, wherein the cascade of temporal stages comprises a dilated convolution stage in series with one or more reweighted dilated convolution stages, and the boundary supervision branch is trained to detect a transition condition between surgical phases; and
performing fusion of an output of the cascade of temporal stages with an output of the boundary supervision branch to predict the surgical phase of the surgical procedure depicted in the frame.
10 . The method of claim 9 , wherein the boundary supervision branch is trained in parallel with the cascade of temporal stages according to a loss function that changes over a plurality of epochs for both the boundary supervision branch and the cascade of temporal stages.
11 . The method of claim 10 , wherein the frame is encoded by an encoder trained using self-supervised learning.
12 . The method of claim 11 , wherein the self-supervised learning comprises a student network that attempts to predict from an augmented image and a target signal generated by a teacher network for a same image under different augmentation.
13 . The method of claim 12 , wherein only the student network is updated through backpropagation, while an exponential moving average is used to update the teacher network, and phase labels are used to update the student network.
14 . The method of claim 13 , wherein a contribution of phase supervision is reduced as training progresses according to a classification loss function.
15 . The method of claim 9 , wherein the video of the surgical procedure is captured by an endoscopic camera from inside of a patient's body.
16 . The method of claim 9 , wherein the video of the surgical procedure is captured by a camera from outside of a patient's body.
17 . A computer program product comprising a non-transitory memory device with computer-readable instructions stored thereon, wherein executing the computer-readable instructions by one or more processing units causes the one or more processing units to perform a plurality of operations comprising:
encoding a frame of a video of a surgical procedure into a plurality of features, wherein the features comprise one or more structures and/or events in the surgical procedure;
providing the features to a boundary supervision branch trained to detect a transition between surgical phases;
providing the features to a cascade of temporal stages of a model configured to adjust a weight for one or more frames based on a confidence score from a previous stage; wherein the cascade of temporal stages comprises a dilated convolution stage in series with one or more reweighted dilated convolution stages; and
performing fusion of an output of the cascade of temporal stages with an output of the boundary supervision branch to predict the surgical phase of the surgical procedure depicted in the frame.
18 . The computer program product of claim 17 , wherein the boundary supervision branch is trained in parallel with the cascade of temporal stages according to a loss function that changes over a plurality of epochs.