Two-stage online detection of action start in untrimmed videos

1. A system for detection of an action start in a video, the system comprising:

a memory storing a plurality of processor-executable instructions; and

a processor executing the plurality of processor-executable instructions to perform operations comprising:

generating, via a classification module, a set of action scores corresponding to a first video frame from the video, based on the first video frame and video frames before the first video frames in the video, each action score indicating a respective probability that the first video frame contains a respective action class;

receiving, at a localization module that is separate from the classification module, the set of action scores from the classification module;

generating, via the localization module, an action-agnostic start probability that the first video frame contains an action start based at least in part on the set of action scores; and

generating, via a fusion component, based on the set of action scores and the action-agnostic start probability, a set of action-specific start probabilities, each action-specific start probability corresponding to a start of an action belonging to the respective action class.

2. The system of claim 1 , wherein the classification module further comprises a deep convolutional network for:

receiving the first video frame from the video; and

generating a feature representation of the first video frame.

3. The system of claim 2 , wherein the classification module further comprises:

a recurrent neural network coupled to the deep convolutional network for:

receiving the feature representation of the first video frame;

generating an updated state based on an input of the feature representation and a previous state of the recurrent neural network.

4. The system of claim 3 , wherein the operations further comprise:

receiving, via a probability computing module couple to the recurrent neural network, the updated state from the recurrent neural network; and

computing, via a softmax operation, the set of action scores based on the updated state and a set of parameters of the classification module.

5. The system of claim 1 , wherein the operations further comprise:

computing, via a loss module, an entropy loss based on set of action scores, wherein the entropy loss is used for updating a set of parameters of the classification module via backpropagation.

6. The system of claim 1 , wherein the localization module further comprises:

a recurrent neural network for:

receiving the set of action scores from the classification module; and

updating a current state based on an input of the set of action scores, a previous state of the recurrent neural network and a set of previously generated action-agnostic start probabilities.

7. The system of claim 6 , wherein the localization module further comprises:

a fully connected layer coupled to the recurrent neural network for computing, via a softmax operation, the action-agnostic start probability based on the current state of the recurrent neural network and a set of parameters of the localization module.

8. The system of claim 6 , wherein the localization module further updates the set of previously generated action-agnostic start probabilities with the generated action-agnostic start probability at a current time instance and

uses the updated set of previously generated action-agnostic start probabilities including the generated action-agnostic start probability as part of the input to the recurrent neural network at a next time instance.

9. The system of claim 6 , wherein the localization module further generates, together with the action-agnostic start probability, a baseline value for a policy gradient update of the localization module.

10. The system of claim 1 , wherein the operations further comprise:

generating, via an action start generation module an action start point of a specific action class in the video based at least in part on a largest value of the set of action-specific start probabilities.

11. A method for detection of an action start in a video, the method comprising:

generating, via a classification module, a set of action scores corresponding to a first video frame from the video, based on the first video frame and video frames before the first video frames in the video,

each action score indicating a respective probability that the first video frame contains a respective action class;

generating, via a localization module that is separate from the classification module, an action-agnostic start probability that the first video frame contains an action start based at least in part on the set of action scores; and

generating, at a fusion component coupled to the localization module and the localization module, based on the set of action scores and the action-agnostic start probability, a set of action-specific start probabilities, each action-specific start probability corresponding to a start of an action belonging to the respective action class.

12. The method of claim 11 , further comprising:

receiving, at the classification module, the first video frame from the video; and

generating a feature representation of the first video frame.

13. The method of claim 12 , further comprising:

receiving, at a recurrent neural network coupled to the deep convolutional network, the feature representation of the first video frame; and

generating an updated state based on an input of the feature representation and a previous state of the recurrent neural network.

14. The method of claim 13 , further comprising:

receiving, at a probability computing module coupled to the recurrent neural network, the updated state from the recurrent neural network; and

computing, via a softmax operation, the set of action scores based on the updated state and a set of parameters of the classification module.

15. The method of claim 11 , further comprising:

computing an entropy loss based on set of action scores, wherein the entropy loss is used for updating a set of parameters of the classification module via backpropagation.

16. The method of claim 11 , further comprising:

receiving, at a recurrent neural network, the set of action scores from the classification module; and

updating a current state based on an input of the set of action scores, a previous state of the recurrent neural network and a set of previously generated action-agnostic start probabilities.

17. The method of claim 16 , further comprising:

computing, at a fully connected layer coupled to the recurrent neural network for computing, via a softmax operation, the action-agnostic start probability based on the current state of the recurrent neural network and a set of parameters of the localization module.

18. The method of claim 16 , further comprising:

updating the set of previously generated action-agnostic start probabilities with the generated action-agnostic start probability at a current time instance; and

using the updated set of previously generated action-agnostic start probabilities including the generated action-agnostic start probability as part of the input to the recurrent neural network at a next time instance.

19. The method of claim 16 , further comprising generating, together with the action-agnostic start probability, a baseline value for a policy gradient update of the localization module.

20. The method of claim 11 , further comprising:

generating an action start point of a specific action class in the video based at least in part on a largest value of the set of action-specific start probabilities.