Weakly supervised action localization
Methods and systems for temporal action localization include processing a video stream to identify an action and a start time and a stop time for the action using a neural network model that separately processes information of appearance and motion modalities from the video stream using transformer branches that include a self-attention and a cross-attention between the appearance and motion modalities. An action is performed responsive to the identified action.
1 . A computer-implemented method for training a neural network model, comprising:
pre-training transformers within a neural network model, wherein the transformers each include a self-attention and a cross-attention between appearance and motion modalities from a video stream; and
tuning the neural network model to perform temporal action localization, using a loss function that includes a mutual learning loss and a pseudo-label loss that considers weakly annotated training data identifying an action within a video without identifying start and stop times of the action, including generating a pseudo-temporal class activation map that includes a matrix having thresholded per-frame classification probabilities for a set of action classes, wherein the pseudo-label loss considers the pseudo-temporal class activation map and an associated uncertainty to mitigate an impact of incorrect pseudo-labels and is expressed as
ℒ
pseudo
=
1
T
∑
t
=
1
T
e
-
u
t
ℒ
ce
(
p
ˆ
t
,
o
t
)
+
τ
u
t
where T is a number of snippets, u t denotes an uncertainty value for a snippet t, ce is a cross-entropy loss term based on snippet-level pseudo-labels {circumflex over (p)} t , o t is a snippet from a temporal class activation map, and τ is hyper-parameter.
2 . The method of claim 1 , wherein the mutual learning loss enforces consistency between appearance attention weights and motion attention weights.
3 . The method of claim 2 , wherein the mutual learning loss is expressed as:
ℒ
ml
=
1
/
2
(
ℒ
m
s
e
(
a
r
,
ϕ
(
a
0
)
)
+
ℒ
mse
(
ϕ
(
a
r
)
,
a
o
)
)
where mse is a mean square loss function based on appearance attention weights a r and motion attention weights a o and ϕ(⋅) is function that truncates a gradient of its input.
4 . The method of claim 1 , wherein the transformers include an appearance transformer that receives appearance features as input and a motion transformer that receives motion features as input.
5 . The method of claim 4 , wherein the cross-attention of the appearance transformer includes key and value terms that are influenced by a query term of the cross-attention of the motion transformer and wherein the cross-attention of the motion transformer includes key and value terms that are influenced by a query term of the cross-attention of the appearance transformer.
6 . The method of claim 1 , wherein the neural network model includes a classifier that generates a temporal class activation map that includes a matrix having per-frame classification probabilities for a set of action classes.
7 . The method of claim 6 , wherein the classifier accepts as input a fused feature that concatenates outputs of the transformer branches.
8 . A computer-implemented method for training a neural network model, comprising:
pre-training transformers within a neural network model, wherein the transformers each include a self-attention and a cross-attention between appearance and motion modalities from a video stream; and
tuning the neural network model to perform temporal action localization, using a loss function that includes a mutual learning loss and a pseudo-label loss that considers weakly annotated training data identifying an action within a video without identifying start and stop times of the action, wherein the mutual learning loss enforces consistency between appearance attention weights and motion attention weights and is expressed as:
ℒ
ml
=
1
/
2
(
ℒ
m
s
e
(
a
r
,
ϕ
(
a
0
)
)
+
ℒ
mse
(
ϕ
(
a
r
)
,
a
o
)
)
where mse is a mean square loss function based on appearance attention weights a r and motion attention weights a o and ϕ(⋅) is function that truncates a gradient of its input.
9 . The method of claim 8 , wherein the transformers include an appearance transformer that receives appearance features as input and a motion transformer that receives motion features as input.
10 . The method of claim 9 , wherein the cross-attention of the appearance transformer includes key and value terms that are influenced by a query term of the cross-attention of the motion transformer and wherein the cross-attention of the motion transformer includes key and value terms that are influenced by a query term of the cross-attention of the appearance transformer.
11 . The method of claim 8 , wherein the neural network model includes a classifier that generates a temporal class activation map that includes a matrix having per-frame classification probabilities for a set of action classes.
12 . The method of claim 11 , wherein the classifier accepts as input a fused feature that concatenates outputs of the transformer branches.