Systems and methods for partially supervised online action detection in untrimmed videos
Embodiments described herein provide systems and methods for a partially supervised training model for online action detection. Specifically, the online action detection framework may include two modules that are trained jointly—a Temporal Proposal Generator (TPG) and an Online Action Recognizer (OAR). In the training phase, OAR performs both online per-frame action recognition and start point detection. At the same time, TPG generates class-wise temporal action proposals serving as noisy supervisions for OAR. TPG is then optimized with the video-level annotations. In this way, the online action detection framework can be trained with video-category labels only without pre-annotated segment-level boundary labels.
1. A method of training an online action detection (OAD) neural network model using a training dataset of untrimmed videos having video-level labels without annotated labels indicating whether a specific video frame contains an action start of a specific action class, the method comprising:
receiving, by a communication interface, an input of the training dataset of untrimmed videos including a set of video-level labels indicating one or more action classes that emerge in the untrimmed videos, wherein the untrimmed videos are used for training the OAD neural network model with no annotated label indicating action starts of the one or more action classes;
generating, by a feature extractor neural network model implemented on one or more hardware processors, feature representations from the training dataset of the untrimmed videos;
generating, by a temporal proposal generator (TPG) neural network model implemented on the one or more hardware processors and receiving the feature representations from the feature extractor neural network, class-wise temporal proposals indicating a respective estimated action start for each action class of the one or more action classes based on the feature representations and the set of video-level labels;
generating, by an online action recognizer (OAR) neural network model implemented on the one or more hardware processors and receiving the feature representations from the feature extractor neural network and the class-wise temporal proposals from the TPG neural network model, per-frame action scores over action classes indicating whether each frame of the untrimmed video contains each specific action class, and a class-agnostic start score indicating whether the respective frame contains a start of any action based on the feature representations and the class-wise temporal proposals;
training the OAD neural network model comprising the TPG neural network model and the OAR neural network model according to a loss metric computed based on the per-frame action scores, the class-agnostic start scores and based on the class-wise temporal proposals as pseudo ground-truth labels; and
generating, by the trained OAD neural network model, a predicted action start for an input real-time video stream.
2. The method of claim 1 , wherein the class-wise temporal proposals indicating the estimated action start for each action class are generated by:
generating, for each untrimmed video with a respective video-level annotation, per-frame action scores for each action class using supervised offline action localization problem; and
computing a video-level score for each action class based on the generated per-frame action scores.
3. The method of claim 2 , further comprising:
selecting a first set of action classes corresponding to the video-level scores higher than a first threshold;
applying a second threshold on a subset of the per-frame scores that corresponds to the first set of action classes along a temporal axis; and
obtaining the class-wise temporal proposals based temporal constraints of the subset of the per-frame scores.
4. The method of claim 2 , further comprising:
computing a predicted video-level probability based on the video-level score for each action class; and
computing a multiple instance learning loss by comparing the respective video-level annotation and the predicted video-level probability.
5. The method of claim 2 , further comprising:
computing a first region feature representation corresponding to a first region of an untrimmed video having a first activity level and a second region feature representation corresponding to a second region of the untrimmed video having a second activity level;
identifying, for the untrimmed video, another untrimmed video that shares a common video-level label for a specific action class with the untrimmed video; and
computing a pair-wise co-activity similarity loss for the untrimmed video and the another untrimmed video based on the first region feature representation and the second region feature representation.
6. The method of claim 5 , wherein the first region feature representation is computed using a temporal attention vector and a vector of the feature representations,
wherein the temporal attention vector is computed by applying temporal softmax over the per-frame action scores.
7. The method of claim 1 , wherein the generating the per-frame action scores and the class-agnostic start score comprises:
updating, at each timestep, a hidden state and a cell state of a long short-term memory based on an input of the feature representations;
applying max pooling on a set of hidden states between a current timestep and a past timestep to obtain an average hidden state;
generating the per-frame action scores based on the hidden state and a first vector of classifier parameters; and
generating the class-agnostic score based on the average hidden state and a second vector of classifier parameters.
8. The method of claim 7 , further comprising:
converting the class-wise temporal proposals to per-frame action labels and binary start labels;
computing a frame loss using a cross entropy loss between the per-frame action labels and the generated per-frame action scores; and
computing a start loss using a focal loss between the binary start labels and the generated class-agnostic score.
9. The method of claim 8 , wherein the loss metric is computed by a weighted sum of an online action recognizer loss and a temporal proposal generator loss,
wherein the online action recognizer loss includes a sum of the frame loss and the start loss, and
wherein the temporal proposal generator loss includes a multiple instance learning loss and a pair-wise co-activity similarity loss.
10. The method of claim 9 , wherein the input of untrimmed videos comprises a first untrimmed video having video-level annotations only, and a second untrimmed video having frame-level annotations, and wherein the OAD neural network model is trained by a combination of the first untrimmed video, and the second untrimmed video supervised by the frame-level annotations.
11. A system for training an online action detection (OAD) neural network model using a training dataset of untrimmed videos having video-level labels without annotated labels indicating whether a specific video frame contains an action start of a specific action class, the system comprising:
a communication interface that receives an input of the training dataset of untrimmed videos including a set of video-level labels indicating one or more action classes that emerge in the untrimmed videos, wherein the untrimmed videos are used for training the OAD neural network model with no annotated label indicating action starts of the one or more action classes;
a memory storing
a feature extractor neural network model, a temporal proposal generator (TPG) neural network model and the OAD neural network model, and a plurality of processor-executable instructions; and
a processor reading from the memory and executing the plurality of processor-executable instructions to:
generate, by a feature extractor neural network model implemented on one or more hardware processors, feature representations from the input of the untrimmed videos;
generate, by a temporal proposal generator (TPG) neural network model implemented on the one or more hardware processors and receiving the feature representations from the feature extractor neural network, class-wise temporal proposals indicating a respective estimated action start for each action class of the one or more action classes based on the feature representations and the set of video-level labels;
generate, by an online action recognizer (OAR) neural network model implemented on the one or more hardware processors and receiving the feature representations from the feature extractor neural network and the class-wise temporal proposals from the TPG neural network model, per-frame action scores over action classes indicating whether each frame of the untrimmed video contains each specific action class, and a class-agnostic start score indicating whether the respective frame contains a start of any action based on the feature representations and the class-wise temporal proposals;
train the OAD neural network model comprising the TPG neural network model and the OAR neural network model according to a loss metric computed based on the per-frame action scores, the class-agnostic start scores and based on the class-wise temporal proposals as pseudo ground-truth labels; and
generate, by the trained OAD neural network model, a predicted action start for an input real-time video stream.
12. The system of claim 11 , wherein the class-wise temporal proposals indicating an estimated action start label for each action class are generated by:
generating, for each untrimmed video with a respective video-level annotation, per-frame action scores for each action class using supervised offline action localization problem; and
computing a video-level score for each action class based on the generated per-frame action scores.
13. The system of claim 12 , wherein the processor is further configured to:
select a first set of action classes corresponding to the video-level scores higher than a first threshold;
apply a second threshold on a subset of the per-frame scores that corresponds to the first set of action classes along a temporal axis; and
obtain the class-wise temporal proposals based temporal constraints of the subset of the per-frame scores.
14. The system of claim 12 , wherein the processor is further configured to:
compute a predicted video-level probability based on the video-level score for each action class; and
compute a multiple instance learning loss by comparing the respective video-level annotation and the predicted video-level probability.
15. The system of claim 12 , wherein the processor is further configured to:
compute a first region feature representation corresponding to a first region of an untrimmed video having a first activity level and a second region feature representation corresponding to a second region of the untrimmed video having a second activity level;
identify, for the untrimmed video, another untrimmed video that shares a common video-level label for a specific action class with the untrimmed video; and
compute a pair-wise co-activity similarity loss for the untrimmed video and the another untrimmed video based on the first region feature representation and the second region feature representation.
16. The system of claim 15 , wherein the first region feature representation is computed using a temporal attention vector and a vector of the feature representations,
wherein the temporal attention vector is computed by applying temporal softmax over the per-frame action scores.
17. The system of claim 11 , wherein the generating the per-frame action scores and the class-agnostic start score comprises:
updating, at each timestep, a hidden state and a cell state of a long short-term memory based on an input of the feature representations;
applying max pooling on a set of hidden states between a current timestep and a past timestep to obtain an average hidden state;
generating the per-frame action scores based on the hidden state and a first vector of classifier parameters; and
generating the class-agnostic score based on the average hidden state and a second vector of classifier parameters.
18. The system of claim 17 , wherein the processor is further configured to:
convert the class-wise temporal proposals to per-frame action labels and binary start labels;
compute a frame loss using a cross entropy loss between the per-frame action labels and the generated per-frame action scores; and
compute a start loss using a focal loss between the binary start labels and the generated class-agnostic score.
19. The system of claim 18 , wherein the loss metric is computed by a weighted sum of an online action recognizer loss and a temporal proposal generator loss,
wherein the online action recognizer loss includes a sum of the frame loss and the start loss, and
wherein the temporal proposal generator loss includes a multiple instance learning loss and a pair-wise co-activity similarity loss.
20. A non-transitory processor-readable storage medium storing a feature extractor neural network model, a temporal proposal generator (TPG) neural network model and the OAD neural network model and processor-executable instructions for training the OAD neural network model using a training dataset of untrimmed videos having video-level labels without annotated labels indicating whether a specific video frame contains an action start of a specific action class, the processor-executable instructions executable by a processor to perform:
receiving, by a communication interface, an input of the training dataset of untrimmed videos including a set of video-level labels indicating one or more action classes that emerge in the untrimmed videos, wherein the untrimmed videos are used for training the OAD neural network model with no annotated label indicating action starts of the one or more action classes;
generating, by a feature extractor neural network model implemented on one or more hardware processors, feature representations from the training dataset of the untrimmed videos;
generating, by a temporal proposal generator (TPG) neural network model implemented on the one or more hardware processors and receiving the feature representations from the feature extractor neural network, class-wise temporal proposals indicating a respective estimated action start for each action class of the one or more action classes based on the feature representations and the set of video-level labels;
generating, by an online action recognizer (OAR) neural network model implemented on the one or more hardware processors and receiving the feature representations from the feature extractor neural network and the class-wise temporal proposals from the TPG neural network model, per-frame action scores over action classes indicating whether each frame of the untrimmed video contains each specific action class, and a class-agnostic start score indicating whether the respective frame contains a start of any action based on the feature representations and the class-wise temporal proposals;
training the OAD neural network model comprising the TPG neural network model and the OAR neural network model according to a loss metric computed based on the per-frame action scores, the class-agnostic start scores and based on the class-wise temporal proposals as pseudo ground-truth labels; and
generating, by the trained OAD neural network model, a predicted action start for an input real-time video stream.