Few-shot video classification
Methods and systems for video processing include enriching an input video feature from an input video frame set using a meta-action bank video sub-actions to generate enriched features. Reinforced image representation is performed using reinforcement learning to compare support image frames and query image frames and determine an importance of the input video frame. A classification is performed on the input video frame based on the importance and the enriched features to generate a label. An action is performed responsive to the generated label.
1 . A computer-implemented method for video processing, comprising:
enriching an input video feature from an input video frame set using a meta-action bank of a plurality of video sub-actions to generate enriched features, wherein enriching the video frame feature is performed as:
f
˜
*
j
s
/
q
=
(
1
-
ρ
)
f
*
j
s
/
q
+
ρ
∑
l
=
1
κ
α
j
l
d
l
e
where p is a parameter balancing between accepting external knowledge with keeping original representations, a is a cosine similarity between the video frame feature f *j s/q , d l e is a projection coefficient on a basis within the meta-action bank e, κ is a number of bases in the meta-action bank, and s/q indicates that the term is treated the same for support frames s and query frames q;
performing reinforced image representation using reinforcement learning to compare support image frames and query image frames and determine an importance of the input video frame;
performing a classification on the input video frame based on the importance and the enriched features to generate a label; and
performing an action responsive to the generated label.
2 . The method of claim 1 , wherein the reinforcement learning uses a policy based on a feature of the input video frame, an aggregated feature of remaining frames of the input video frame set, and a separate video.
3 . The method of claim 2 , wherein the aggregated feature is a weighted average of features of the remaining frames.
4 . The method of claim 1 , wherein the meta-action bank includes information for a plurality of different action categories to establish an action space.
5 . The method of claim 1 , wherein the label identifies an action within the input video frame set.
6 . The method of claim 1 , wherein performing the action includes an action selected from the group consisting of generating additional information relating to the label and performing a security action relating to the label.
7 . A system for video processing, comprising:
a hardware processor; and
a memory that stores a computer program which, when executed by the hardware processor, causes the hardware processor to:
enrich an input video feature from an input video frame set using a meta-action bank of a plurality of video sub-actions to generate enriched features, wherein enrichment of the video frame feature is performed as:
f
˜
*
j
s
/
q
=
(
1
-
ρ
)
f
*
j
s
/
q
+
ρ
∑
l
=
1
κ
α
j
l
d
l
e
where p is a parameter balancing between accepting external knowledge with keeping original representations, a is a cosine similarity between the video frame feature f *j s/q , d l e is a projection coefficient on a basis within the meta-action bank e, κ is a number of bases in the meta-action bank, and s/q indicates that the term is treated the same for support frames s and query frames q;
perform reinforced image representation using reinforcement learning to compare support image frames and query image frames and determine an importance of the input video frame;
perform a classification on the input video frame based on the importance and the enriched features to generate a label; and
perform an action responsive to the generated label.
8 . The system of claim 7 , wherein the reinforcement learning uses a policy based on a feature of the input video frame, an aggregated feature of remaining frames of the input video frame set, and a separate video.
9 . The system of claim 8 , wherein the aggregated feature is a weighted average of features of the remaining frames.
10 . The system of claim 7 , wherein the meta-action bank includes information for a plurality of different action categories to establish an action space.
11 . The system of claim 7 , wherein the label identifies an action within the input video frame set.
12 . The system of claim 7 , wherein the computer program further causes the hardware processor to perform the action as an action selected from the group consisting of generating additional information relating to the label and performing a security action relating to the label.