Active sparse labeling of video frames
View Patent ↗An active sparse labeling system that provides high performance and low annotation costs by performing partial instance annotation (i.e., sparse labeling) by frame level selection to annotate the most informative frames, thereby improving action detection task efficiencies. The active sparse labeling system utilizes a frame level cost estimation to determine the utility of each frame in a video based on the frame's impact on action detection. The system includes an adaptive proximity-aware uncertainty model, which is an uncertainty-based frame scoring mechanism. The adaptive proximity-aware uncertainty model estimates a frame's utility using the uncertainty of detections of the frame's proximity to existing annotations, thereby determining a diverse set of frames in a video which are effective for learning the task of dense video understanding (such as action detection). In addition, the active sparse labeling system includes a loss formulation training model (max-Gaussian weighted loss) that uses weighted pseudo-labeling.
1 . A system for active sparse labeling of multimedia input to facilitate dense video understanding, comprising at least one computer processor configured to:
a) receive a multimedia input comprising a plurality of frames;
b) apply an adaptive proximity-aware uncertainty selection model to iteratively select a subset of frames from the multimedia input, wherein the selection model is configured to:
i. calculate frame-level uncertainty for each frame based on pixel-wise confidence scores of localization;
ii. determine an adaptive distance metric based on the proximity of a new frame to previously selected frames;
iii. compute a selection score for each frame based on the frame-level uncertainty and the adaptive distance metric;
iv. select frames for annotation when their selection scores exceed a predefined threshold, thereby ensuring a diverse and informative set of frames is chosen across the temporal domain of the multimedia input;
c) annotate the selected subset of frames to create a labeled dataset;
d) use the labeled dataset to train an action detection model to achieve or exceed a predetermined precision benchmark; and
e) update the action detection model iteratively by reapplying the adaptive proximity-aware uncertainty selection model to further refine frame selection based on updated model insights and annotations until the precision benchmark is met or an annotation cost budget is exhausted.
2 . The system of claim 1 , wherein the precision benchmark algorithm is selected from the group consisting of video-metric average precision (v-mAP), frame-metric average precision (f-mAP) and mean average precision (mAP).
3 . The system of claim 1 , wherein the max-Gaussian weighted loss model is further configured to assign weights to each frame based on the proximity of their localization to ground-truth annotations, whereby frames closer to the ground-truth data are given higher weight.
4 . The system of claim 3 , wherein the max-Gaussian weighted loss model adjusts the localization loss for each frame based on the assigned weights.
5 . The system of claim 4 , wherein the localization loss adjustments are applied iteratively, with each iteration of the training cycle using the updated frame weights to continuously refine the action detection model's accuracy.
6 . The system of claim 3 , wherein the weights assigned by the max-Gaussian weighted loss model follow a Gaussian distribution centered on the frame's distance to the nearest ground-truth annotation, with a variance that adjusts adaptively based on the overall performance of the action detection model in previous training iterations.
7 . The system of claim 6 , wherein the variance of the Gaussian distribution used to assign weights to frames is reduced as the difference between the action detection model's performance and the predetermined precision benchmark decreases, thus focusing training more on frames that significantly deviate from desired outcomes.
8 . The system of claim 1 , wherein the adaptive proximity-aware uncertainty selection model incorporates an intra-sample approach whereby the subset of frames selected represents different temporal segments of the multimedia input.
9 . The system of claim 1 , wherein the adaptive proximity-aware uncertainty selection model employs an uncertainty-based scoring mechanism that assigns priority to frames to those with higher uncertainty scores for subsequent annotation.
10 . The system of claim 1 , further comprising a pseudo-label generation function configured to:
a. generate pseudo-labels for non-selected frames utilizing interpolation and a spatio-temporal superpixel method; and
b. refine pseudo-labels over time responsive to newly acquired annotations and prior predictions.
11 . The system of claim 10 , wherein the spatio-temporal superpixel method aggregates visually similar pixels across frames to extend sparse annotations.
12 . The system of claim 1 , further comprising a scoring mechanism configured to compute a selection score for each frame by applying proximity and uncertainty metrics for targeted frame selection.
13 . The system of claim 1 , wherein the system applies a mix of annotation types including bounding boxes, pixel-wise masks, and scribbles.
14 . The system of claim 1 , wherein the computer processor is configured to apply the adaptive proximity-aware uncertainty selection model iteratively across the multimedia input.
15 . A system for active sparse labeling of multimedia input to facilitate dense video understanding, the system comprising at least one computer processor configured to:
a. receive a multimedia input comprising a plurality of frames;
b. apply an adaptive proximity-aware uncertainty selection model to iteratively select a subset of frames from the multimedia input, the model configured to:
i. calculate frame-level uncertainty for each frame based on pixel-wise confidence scores of localization;
ii. determine an adaptive distance metric based on the proximity of a new frame to previously selected frames;
iii. compute a selection score for each frame based on the frame-level uncertainty and the adaptive distance metric;
iv. select frames for annotation when their selection scores exceed a predefined threshold, ensuring a diverse and informative set of frames is chosen across the temporal domain of the multimedia input;
c. annotate the selected subset of frames to create a labeled dataset;
d. use the labeled dataset to train an action detection model to achieve or exceed a predetermined precision benchmark;
e. update the action detection model iteratively by reapplying the adaptive proximity-aware uncertainty selection model to further refine frame selection based on updated model insights and annotations until the precision benchmark is met or an annotation cost budget is exhausted;
f. employ an uncertainty-based scoring mechanism that assigns priority to frames with higher uncertainty scores for subsequent annotation;
g. generate pseudo-labels for non-selected frames utilizing interpolation and a spatio-temporal superpixel method and refine pseudo-labels over time responsive to newly acquired annotations and prior predictions;
h. assign weights to each frame based on the proximity of their localization to ground-truth annotations, whereby frames closer to the ground-truth data are given higher weight;
i. adjust the localization loss for each frame based on the assigned weights; and
j. apply the adaptive proximity-aware uncertainty selection model iteratively across the multimedia input.
16 . A method for active sparse labeling of multimedia input to facilitate dense video understanding, implemented by at least one computer processor, the method comprising:
a. receiving a multimedia input comprising a plurality of frames;
b. applying an adaptive proximity-aware uncertainty selection model to iteratively select a subset of frames from the multimedia input, the model configured to:
i. calculate frame-level uncertainty for each frame based on pixel-wise confidence scores of localization;
ii. determine an adaptive distance metric based on the proximity of a new frame to previously selected frames;
iii. compute a selection score for each frame based on the frame-level uncertainty and the adaptive distance metric;
iv. select frames for annotation when their selection scores exceed a predefined threshold, ensuring a diverse and informative set of frames is chosen across the temporal domain of the multimedia input;
c. annotating the selected subset of frames to create a labeled dataset;
d. using the labeled dataset to train an action detection model to achieve or exceed a predetermined precision benchmark; and
e. updating the action detection model iteratively by reapplying the adaptive proximity-aware uncertainty selection model to further refine frame selection based on updated model insights and annotations until the precision benchmark is met or an annotation cost budget is exhausted.
17 . The method of claim 16 , wherein the precision benchmark algorithm is selected from the group consisting of video-metric average precision (v-mAP), frame-metric average precision (f-mAP) and mean average precision (mAP).
18 . The method of claim 16 , further comprising assigning weights to each frame based on the proximity of their localization to ground-truth annotations, whereby frames closer to the ground-truth data are given higher weight.
19 . The method of claim 18 , further comprising adjusting the localization loss for each frame based on the assigned weights.
20 . The method of claim 19 , wherein the localization loss adjustments are applied iteratively, with each iteration of the training cycle using the updated frame weights to continuously refine the action detection model's accuracy.
21 . The method of claim 18 , wherein the weights assigned by a max-Gaussian weighted loss model follow a Gaussian distribution centered on the frame's distance to the nearest ground-truth annotation, with a variance that adjusts adaptively based on the overall performance of the action detection model in previous training iterations.
22 . The method of claim 21 , wherein the variance of the Gaussian distribution used to assign weights to frames is reduced as the difference between the action detection model's performance and the predetermined precision benchmark decreases, thus focusing training more on frames that significantly deviate from desired outcomes.
23 . The method of claim 16 , wherein the adaptive proximity-aware uncertainty selection model incorporates an intra-sample approach whereby the subset of frames selected represents different temporal segments of the multimedia input.
24 . The method of claim 16 , wherein the adaptive proximity-aware uncertainty selection model employs an uncertainty-based scoring mechanism that assigns priority to frames with higher uncertainty scores for subsequent annotation.
25 . The method of claim 16 , further comprising:
a. generating pseudo-labels for non-selected frames utilizing interpolation and a spatio-temporal superpixel method; and
b. refining pseudo-labels over time responsive to newly acquired annotations and prior predictions.
26 . The method of claim 25 , wherein the spatio-temporal superpixel method aggregates visually similar pixels across frames to extend sparse annotations.
27 . The method of claim 16 , further comprising computing a selection score for each frame by applying proximity and uncertainty metrics for targeted frame selection.
28 . The method of claim 16 , wherein a mix of annotation types including bounding boxes, pixel-wise masks, and scribbles is applied.
29 . The method of claim 16 , wherein the adaptive proximity-aware uncertainty selection model is applied iteratively across the multimedia input.