System and method for adverse event detection or severity estimation from surgical data
Embodiments described herein may provide devices, systems, methods, and/or computer readable medium for adverse event detection and severity estimation in surgical videos. The system can train multiple models for adverse detection and severity estimation. The system can load selected models for real-time adverse event detection and severity estimation.
1. A system for automatically generating data structures configured to store classifications relating to an adverse event based on audio or video data, the classifications based at least on a plurality of classification tasks for injury event detection, the system comprising:
a processor, operating in conjunction with computer memory, the processor configured to:
receive a set of audio or video data;
extract, using a feature extractor neural network, a vector of latent features from the set of audio or video data;
provide, to each of a plurality of time-based classifiers, the vector of latent features from the feature extractor neural network, each time-based classifier corresponding to a classification task of the plurality of classification tasks configured to detect an injury event;
train the feature extractor neural network on a training data set using a sigmoid binary cross-entropy loss and a regression loss; and
train each time-based classifier of the plurality of time-based classifiers separately on each classification task of the plurality of classification tasks with a loss function that includes at least the sigmoid binary cross-entropy loss and the regression loss;
wherein the regression loss is a truncated mean-square error (T-MSE) that minimizes a number of transitions from one action to another, and the combination of the regression loss and the sigmoid binary cross-entropy loss adapts the feature extractor neural network for multi-task event detection;
wherein the loss function is optimized for event-wise metric performance, based at least on intersection over union thresholds, over frame-wise metric performance by (1) incorporating a focal loss term to address class imbalance by scaling the loss function for each time-based classifier by a factor of 1−p, where p is a confidence value of a prediction, and (2) incorporating an uncertainty loss in the loss function for each time-based classifier by adding a trainable weight σ 2 c to each task, the trainable weight representing a learnable scalar.
2. The system of claim 1 , wherein the set of audio or video data includes a set of video frames that have been stabilized to reduce camera motion through the use of bundled-camera path stabilization that reduces jitter and smooths camera paths so that the latent features are accumulated across a plurality of frames.
3. The system of claim 2 , wherein stabilization includes warping images to align each frame's camera view based at least on homography.
4. The system of claim 1 , wherein the feature extractor neural network is a three dimensional (3D) or two-dimensional (2D) convolutional network.
5. The system of claim 1 , wherein the classification tasks include at least bleeding and thermal injury detection, and wherein the classification tasks are causally distinct and include distinguishing active injury events from prior injury artifacts.
6. The system of claim 1 , wherein the loss function for each time-based classifier further includes both focal and the uncertainty loss wherein the loss function is based on the relation:
ℒ
focal
‐
uncertainty
=
(
1
CN
∑
c
∑
n
1
σ
c
2
(
1
-
p
cn
)
ℒ
BCE
cn
+
log
σ
c
)
+
λℒ
T
‐
MSE
;
where C is a number of classes, N is a number of samples, A is a smoothing loss constant, L BCEcn is a corresponding binary cross entropy of class c and sample n, p cn is a confidence probability of a prediction of class c at sample n, and σ 2 c is the learnable scalar added from the uncertainty loss.
7. The system of claim 1 , wherein the processor is configured to receive a set of audio data, and the feature extractor neural network extracts the vector of latent features from a combination of the set of audio data and the set of video data.
8. The system of claim 7 , wherein the training data set includes both training video data and training audio data.
9. A method for automatically generating data structures configured to store classifications relating to an adverse event based on audio or video data, the classifications based at least on a plurality of classification tasks, the method comprising:
receiving a set of audio or video data;
extracting, using a feature extractor neural network, a vector of latent features from the set of audio or video data;
providing, to each of a plurality of time-based classifiers, the vector of latent features from the feature extractor neural network, each time-based classifier corresponding to a classification task of the plurality of classification tasks configured to detect an injury event;
training the feature extractor neural network on a training data set using a sigmoid binary cross-entropy loss and a regression loss; and
training each time-based classifier of the plurality of time-based classifiers separately on each classification task of the plurality of classification tasks with a loss function that includes at least the sigmoid binary cross-entropy loss and the regression loss;
wherein the regression loss is a truncated mean-square error (T-MSE) that minimizes a number of transitions from one action to another, and the combination of the regression loss and the sigmoid binary cross-entropy loss adapts the feature extractor neural network for multi-task event detection;
wherein the loss function is optimized for event-wise metric performance, based at least on intersection over union thresholds, over frame-wise metric performance by (1) incorporating a focal loss term to address class imbalance by scaling the loss function for each time-based classifier by a factor of 1−p, where p is a confidence value of a prediction, and (2) incorporating an uncertainty loss in the loss function for each time-based classifier by adding a trainable weight σ 2 c to each task, the trainable weight representing a learnable scalar.
10. The method of claim 9 , wherein the set of audio or video data includes a set of video frames that have been stabilized to reduce camera motion through the use of bundled-camera path stabilization that reduces jitter and smooths camera paths so that the latent features are accumulated across a plurality of frames.
11. The method of claim 10 , wherein stabilization includes warping images to align each frame's camera view based at least on homography.
12. The method of claim 9 , wherein the feature extractor neural network is a three dimensional (3D) or two-dimensional (2D) convolutional network.
13. The method of claim 9 , wherein the classification tasks include at least bleeding and thermal injury detection, and wherein the classification tasks are causally distinct and include distinguishing active injury events from prior injury artifacts.
14. The method of claim 9 , wherein the loss function for each time-based classifier further includes both focal and the uncertainty loss, wherein the loss function is based on the relation:
ℒ
focal
‐
uncertainty
=
(
1
CN
∑
c
∑
n
1
σ
c
2
(
1
-
p
cn
)
ℒ
BCE
cn
+
log
σ
c
)
+
λℒ
T
‐
MSE
;
where C is a number of classes, N is a number of samples, A is a smoothing loss constant, L BCEcn is a corresponding binary cross entropy of class c and sample n, p cn is a confidence probability of a prediction of class c at sample n, and σ 2 c is the learnable scalar added from the uncertainty loss.
15. The method of claim 9 , the method comprising receiving a set of audio data, and extracting, by the feature extractor neural network, the vector of latent features from a combination of the set of audio data and the set of video data.
16. A non-transitory computer readable medium storing machine interpretable instructions, the machine interpretable instructions, which when executed by a processor, cause the processor to perform a method for automatically generating data structures configured to store classifications relating to an adverse event based on audio or video data, the classifications based at least on a plurality of classification tasks, the method comprising:
receiving a set of audio or video data;
extracting, using a feature extractor neural network, a vector of latent features from the set of audio or video data;
providing, to each of a plurality of time-based classifiers, the vector of latent features from the feature extractor neural network, each time-based classifier corresponding to a classification task of the plurality of classification tasks configured to detect an injury event;
training the feature extractor neural network on a training data set using a sigmoid binary cross-entropy loss and a regression loss; and
training each time-based classifier of the plurality of time-based classifiers separately on each classification task of the plurality of classification tasks with a loss function that includes at least the sigmoid binary cross-entropy loss and the regression loss;
wherein the regression loss is a truncated mean-square error (T-MSE) that minimizes a number of transitions from one action to another, and the combination of the regression loss and the sigmoid binary cross-entropy loss adapts the feature extractor neural network for multi-task event detection;
wherein the loss function is optimized for event-wise metric performance, based at least on intersection over union thresholds, over frame-wise metric performance by (1) incorporating a focal loss term to address class imbalance by scaling the loss function for each time-based classifier by a factor of 1−p, where p is a confidence value of a prediction, and (2) incorporating an uncertainty loss in the loss function for each time-based classifier by adding a trainable weight σ 2 c to each task, the trainable weight representing a learnable scalar.