Framework for few-shot temporal action localization
Systems and techniques that facilitate few-shot temporal action localization based on graph convolutional networks are provided. In one or more embodiments, a graph component can generate a graph that models a support set of temporal action classifications. Nodes of the graph can correspond to respective temporal action classifications in the support set. Edges of the graph can correspond to similarities between the respective temporal action classifications. In various embodiments, a convolution component can perform a convolution on the graph, such that the nodes of the graph output respective matching scores indicating levels of match between the respective temporal action classifications and an action to be classified. In various embodiments, an instantiation component can input into the nodes respective input vectors based on a proposed feature vector representing the action to be classified. In various cases, the respective temporal action classifications can correspond to respective example feature vectors, and the respective input vectors can be concatenations of the respective example feature vectors and the proposed feature vector.
1. A system, comprising:
a memory that stores computer-executable components; and
a processor, operably coupled to the memory, that executes the computer-executable components stored in the memory, wherein the computer-executable components comprise:
a graph component that generates a graph modeling a support set of temporal action classifications, such that nodes of the graph correspond to respective temporal action classifications in the support set, and such that edges of the graph correspond to similarities between the respective temporal action classifications; and
a convolution component that performs a convolution on the graph, such that the nodes output respective matching scores indicating levels of match between the respective temporal action classifications and an action to be classified.
2. The system of claim 1 , further comprising:
an instantiation component that inputs into the nodes respective input vectors based on a proposed feature vector representing the action to be classified.
3. The system of claim 2 , wherein
the respective temporal action classifications correspond to respective example feature vectors; and
the respective input vectors are concatenations of the respective example feature vectors and the proposed feature vector representing the action to be classified.
4. The system of claim 3 , wherein:
the respective example feature vectors are generated via a first gated recurrent unit based on respective one-shot support videos that display exemplars of the respective temporal action classifications; and
the proposed feature vector is generated via a second gated recurrent unit based on an untrimmed input video that displays the action to be classified.
5. The system of claim 3 , wherein the similarities are calculated based on cosine similarities between respective example feature vectors of respective temporal action classifications.
6. The system of claim 1 , wherein the action to be classified is determined to belong to a temporal action classification having a highest matching score.
7. The system of claim 1 , wherein the graph is an undirected graph.
8. A computer-implemented method, comprising:
generating, by a device operatively coupled to a processor, a graph modeling a support set of temporal action classifications, such that nodes of the graph correspond to respective temporal action classifications in the support set, and such that edges of the graph correspond to similarities between the respective temporal action classifications; and
performing, by the device, a convolution on the graph, such that the nodes output respective matching scores indicating levels of match between the respective temporal action classifications and an action to be classified.
9. The computer-implemented method of claim 8 , further comprising:
inputting, by the device, into the nodes respective input vectors based on a proposed feature vector representing the action to be classified.
10. The computer-implemented method of claim 9 , wherein
the respective temporal action classifications correspond to respective example feature vectors; and
the respective input vectors are concatenations of the respective example feature vectors and the proposed feature vector representing the action to be classified.
11. The computer-implemented method of claim 10 , further comprising:
generating, by the device, the respective example feature vectors via a first gated recurrent unit based on respective one-shot support videos that display exemplars of the respective temporal action classifications; and
generating, by the device, the proposed feature vector via a second gated recurrent unit based on an untrimmed input video that displays the action to be classified.
12. The computer-implemented method of claim 10 , wherein the similarities are calculated based on cosine similarities between respective example feature vectors of respective temporal action classifications.
13. The computer-implemented method of claim 8 , wherein the action to be classified is determined to belong to a temporal action classification having a highest matching score.
14. The computer-implemented method of claim 8 , wherein the graph is an undirected graph.
15. A computer program product for facilitating few-shot action localization, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processing component to cause the processing component to:
generate a graph modeling a support set of temporal action classifications, such that nodes of the graph correspond to respective temporal action classifications in the support set, and such that edges of the graph correspond to similarities between the respective temporal action classifications; and
perform a convolution on the graph, such that the nodes output respective matching scores indicating levels of match between the respective temporal action classifications and an action to be classified.
16. The computer program product of claim 15 , wherein the program instructions are further executable to cause the processing component to:
input into the nodes respective input vectors based on a proposed feature vector representing the action to be classified.
17. The computer program product of claim 16 , wherein
the respective temporal action classifications correspond to respective example feature vectors; and
the respective input vectors are concatenations of the respective example feature vectors and the proposed feature vector representing the action to be classified.
18. The computer program product of claim 17 , wherein the program instructions are further executable to cause the processing component to:
generate the respective example feature vectors via a first gated recurrent unit based on respective one-shot support videos that display exemplars of the respective temporal action classifications; and
generate the proposed feature vector via a second gated recurrent unit based on an untrimmed input video that displays the action to be classified.
19. The computer program product of claim 17 , wherein the similarities are calculated based on cosine similarities between respective example feature vectors of respective temporal action classifications.
20. The computer program product of claim 15 , wherein the action to be classified is determined to belong to a temporal action classification having a highest matching score.