Permutation invariant convolution (PIC) for recognizing long-range activities, generating global representation of input streams and classifying activity based on global representation
A method for recognizing long-range activities in videos includes segmenting an input video stream to generate multiple frame sets. For each of the frame sets, a frame with a highest likelihood of including one or more actions of a set of predefined actions is identified regardless of its order in the frame set. A global representation of the input stream is generated based on pooled representations of the identified frames. A long-range activity in the video stream is classified based on the global representation.
1. A method, comprising:
segmenting an input stream to generate a plurality of frame sets, each frame set including a plurality of frames;
identifying, by a permutation invariant convolutional layer of a neural network, for each frame set from the plurality of frame sets, a frame with a highest likelihood of including one or more actions of a set of predefined actions;
generating a global representation of the input stream from pooled representations of the identified frames; and
classifying a long-range activity based on the global representation.
2. The method of claim 1 , in which identifying the frame comprises generating a similarity matrix from a dot product of features of the frame set and a first kernel.
3. The method of claim 2 , further comprising max-pooling similarities in the similarity matrix to identify the frame with the highest likelihood.
4. The method of claim 3 , further comprising:
generating an attention vector from the similarity matrix; and
generating the global representation based on a dot product of the attention vector and a second kernel.
5. The method of claim 4 , in which the first kernel and the second kernel are linked.
6. The method of claim 1 , in which the global representation is based on a dot product of the pooled representations of the identified frames.
7. The method of claim 1 , in which generating the global representation is performed by the permutation invariant convolutional layer of the neural network.
8. The method of claim 7 , in which the neural network comprises a plurality of cascading permutation invariant convolutional layers.
9. An apparatus, comprising:
a memory; and
at least one processor coupled to the memory, the at least one processor configured to:
segment an input stream to generate a plurality of frame sets, each frame set including a plurality of frames;
identify, by a permutation invariant convolutional layer of a neural network, for each frame set from the plurality of frame sets, a frame with a highest likelihood of including a one or more actions of a set of predefined actions;
generate a global representation of the input stream from pooled representations of the identified frames; and
classify a long-range activity based on the global representation.
10. The apparatus of claim 9 , in which the at least one processor is configured to identify the frame by generating a similarity matrix from a dot product of features of the frame set and a first kernel.
11. The apparatus of claim 10 , in which the at least one processor is further configured to max-pool similarities in the similarity matrix to identify the frame with the highest likelihood.
12. The apparatus of claim 11 , in which the at least one processor is further configured to:
generating an attention vector from the similarity matrix; and
generating the global representation based on a dot product of the attention vector and a second kernel.
13. The apparatus of claim 12 , in which the first kernel and the second kernel are linked.
14. The apparatus of claim 9 , in which the at least on processor is further configured to generate the global representation based on a dot product of the pooled representations of the identified frames.
15. The apparatus of claim 9 , in which the at least one processor is further configured to generate the global representation by the permutation invariant convolutional layer of the neural network.
16. The apparatus of claim 15 , in which the neural network comprises a plurality of cascading permutation invariant convolutional layers.
17. An apparatus, comprising:
means for segmenting an input stream to generate a plurality of frame sets, each frame set including a plurality of frames;
means for identifying, by a permutation invariant convolutional layer of a neural network, for each frame set from the plurality of frame sets, a frame with a highest likelihood of including one or more actions of a set of predefined actions;
means for generating a global representation of the input stream from pooled representations of the identified frames; and
means for classifying a long-range activity based on the global representation.
18. The apparatus of claim 17 , further comprising means for generating a similarity matrix from a dot product of features of the frame set and a first kernel.
19. The apparatus of claim 18 , further comprising means for max-pooling similarities in the similarity matrix to identify the frame with the highest likelihood.
20. The apparatus of claim 19 , further comprising:
means for generating an attention vector from the similarity matrix; and
means for generating the global representation based on a dot product of the attention vector and a second kernel.
21. The apparatus of claim 20 , in which the first kernel and the second kernel are linked.
22. The apparatus of claim 17 , in which the global representation is based on a dot product of the pooled representations of the identified frames.
23. A non-transitory computer-readable medium having program code recorded, the program code executed by a processor and comprising:
program code to segment an input stream to generate a plurality of frame sets, each frame set including a plurality of frames;
program code to identify, by a permutation invariant convolutional layer of a neural network, for each frame set from the plurality of frame sets, a frame with a highest likelihood of including one or more actions of a set of predefined actions;
program code to generate a global representation of the input stream from pooled representations of the identified frames; and
program code to classify a long-range activity based on the global representation.
24. The non-transitory computer-readable medium of claim 23 , further comprising program code to identify the frame by generating a similarity matrix from a dot product of features of the frame set and a first kernel.
25. The non-transitory computer-readable medium of claim 24 , further comprising program code to max-pool similarities in the similarity matrix to identify the frame with the highest likelihood.
26. The non-transitory computer-readable medium of claim 25 , further comprising program code:
program code to generate an attention vector from the similarity matrix; and
program code to generate the global representation based on a dot product of the attention vector and a second kernel.
27. The non-transitory computer-readable medium of claim 26 , in which the first kernel and the second kernel are linked.
28. The non-transitory computer-readable medium of claim 23 , further comprising program code to generate the global representation based on a dot product of the pooled representations of the identified frames.
29. The non-transitory computer-readable medium of claim 23 , further comprising program code to generate the global representation at the permutation invariant convolutional layer of the neural network.
30. The non-transitory computer-readable medium of claim 29 , in which the neural network comprises a plurality of cascading permutation invariant convolutional layers.