Systems, methods and computer media for joint attention video processing

1. A computer implemented method for processing a video, the method comprising:

receiving a plurality of video frames of the video;

generating a plurality of first input features based on the plurality of video frames;

generating a plurality of second input features based on reversing a temporal order of the plurality of first input features;

generating a first set of joint attention features based on the plurality of first input features;

generating a second set of joint attention features based on the plurality of second input features; and

concatenating the first set of joint attention features and the second set of joint attention features to generate a final set of joint attention features.

2. The method of claim 1 , wherein generating the first set of joint attention features based on the plurality of first input features comprises:

generating a set of attention weights based on the plurality of first input features;

extracting a set of position-encoded input features based on the plurality of first input features; and

computing the first set of joint attention features based on multiplying the set of attention weights by the set of position-encoded input features.

3. The method of claim 1 , wherein generating the set of attention weights for the plurality of first input features comprises:

adding a dimension to the plurality of first input features to generate a 2D feature map; and

transforming the 2D feature map to generate the set of attention weights for the plurality of first input features.

4. The method of claim 3 , further comprising:

processing the final set of joint attention features to generate:

one or more action localization labels of the video;

a corresponding start time for each of the one or more action localization labels of the video; and

a corresponding end time for each of the one or more action localization labels of the video.

5. The method of claim 4 , wherein the one or more action localization labels of the video are generated by a Boundary Matching Network (BMN) or a Sub-Graph Temporal Action Detection (GTAD) based on the final set of joint attention features.

6. The method of claim 1 , wherein the plurality of first input features are generated based on the plurality of video frames by an action classification model.

7. The method of claim 6 , wherein the action classification model comprises Temporal Segment Networks (TSN) or a Temporal Shift Model (TSM).

8. A computer implemented method for training an action classification model, the method comprising, iteratively, for each video clip in a plurality of video clips of a video:

receiving a plurality of video frames of the video clip;

retrieving a plurality of first input features previously extracted based on the plurality of video frames;

generating a plurality of second input features, by the action classification model, based on the plurality of video frames;

computing a classification loss of the action classification model based on the plurality of second input features;

generating an updated set of input features based on the plurality of first input features and the plurality of second input features;

generating a plurality of action localization labels using an action localization network based on the updated set of input features;

computing a localization loss of the action localization network based on the plurality of action localization labels;

computing an updated loss based on the classification loss of the action classification model and the localization loss of the action localization network; and

adjusting one or more weights of the action classification model based on the updated loss.

9. The method of claim 8 , wherein the updated set of input features is generated based on the equation:

C i =α*C i,new +(1−α)* C i,old

wherein C i represents the updated set of input features, C i,old represents the plurality of first input features, C i,new represents the plurality of second input features, and α represents a parameter ϵ R.

10. The method of claim 9 , wherein α has a value between 0.01 to 0.1.

11. The method of claim 9 , wherein generating the plurality of action localization labels using the action localization network based on the updated set of input features comprises:

retrieving a set of previously extracted input features C V of the video, C V including C i,old ;

updating C V by replacing C i,old with C i to obtain updated C V ′; and

computing the plurality of action localization labels using the action localization network based on the updated C V ′.

12. The method of claim 8 , wherein the updated loss is computed based on the equation:

L T =β*L c +(1−β)* L l

wherein L T represents the updated loss, L c represents the classification loss of the action classification model, L l represents the localization loss of the action localization network, and β represents a parameter ϵ R.

13. The method of claim 12 , wherein β has a value between 0.1 to 0.9.

14. The method of claim 8 , wherein the action classification model comprises Temporal Segment Networks or a Temporal Shift Model.

15. The method of claim 8 , wherein the action localization network comprises a BMN, a Gaussian Temporal Awareness Network or a GTAD.

16. A system for processing a video, the system comprising:

a processing unit; and

a memory coupled to the processing unit, the memory storing machine-executable instructions that, when executed by the processing unit, cause the system to:

receive a plurality of video frames of a video;

generate a plurality of first input features based on the plurality of video frames;

generate a plurality of second input features based on reversing a temporal order of the plurality of first input features;

generate a first set of joint attention features based on the plurality of first input features;

generate a second set of joint attention features based on the plurality of second input features; and

concatenate the first set of joint attention features and the second set of joint attention features to generate a final set of joint attention features.

17. The system of claim 16 , wherein the machine-executable instructions, when executed by the processing unit, cause the system to generate the first set of joint attention features by:

generating a set of attention weights based on the plurality of first input features;

extracting a set of position-encoded input features based on the plurality of first input features; and

computing the first set of joint attention features based on multiplying the set of attention weights by the set of position-encoded input features.

18. The system of claim 16 , wherein the machine-executable instructions, when executed by the processing unit, cause the system to generate the set of attention weights for the plurality of first input features by:

adding a dimension to the plurality of first input features to generate a 2D feature map; and

transforming the 2D feature map to generate the set of attention weights for the plurality of first input features.

19. The system of claim 18 , wherein the machine-executable instructions, when executed by the processing unit, further cause the system to:

process the final set of joint attention features to generate:

one or more action localization labels of the video;

a corresponding start time for each of the one or more action localization labels of the video; and

a corresponding end time for each of the one or more action localization labels of the video.

20. The system of claim 16 , wherein the plurality of first input features are generated based on the plurality of video frames by an action classification model.