Spatio-temporal interaction network for learning object interactions

1. A computer-implemented method for improving video understanding tasks based on higher-order object interactions (HOIs) between object features, comprising:

obtaining a plurality of frames of a video;

generating a coarse-grained feature representation, including generating an image feature for each of a plurality of timesteps respectively corresponding to each of the frames, and performing attention based on the image features;

generating a fine-grained feature representation, including generating an object feature for each of the plurality of timesteps, and generating the HOIs between the object features; and

concatenating the coarse-grained and the fine-grained feature representations to generate a concatenated feature representation.

2. The method of claim 1 , further comprising making a prediction based on the concatenated feature representation to perform one or more operations.

3. The method of claim 2 , wherein the one or more operations include at least one of action recognition and video captioning.

4. The method of claim 1 , wherein the image features are generated using a convolutional neural network (CNN).

5. The method of claim 1 , wherein performing attention based on the image features further includes performing Scaled Dot Product (SDP) attention.

6. The method of claim 1 , wherein the coarse-grained feature representation includes attended context information.

7. The method of claim 6 , wherein generating the coarse-grained feature representation further comprises generating the attended context information by mean-pooling weighted image representations.

8. The method of claim 1 , wherein the object features are generated using a region proposal network (RPN).

9. The method of claim 1 , wherein generating the HOIs further includes:

generating projected object features respectively corresponding to the object features;

computing attention weights based at least in part on the projected object features;

performing attentive selection using a plurality of attentive selection components to generate attended object features each corresponding to a respective one of the attentive selection components;

concatenating the attended object features to generate an attended object feature representation; and

determining an HOI representation based on the attended object feature representation.

10. The method of claim 9 , wherein the projected object features are generated using a plurality of multilayer perceptrons (MLPs).

11. The method of claim 9 , wherein the attentive selection components implement dot product attention.

12. The method of claim 9 , wherein determining the HOI representation further includes implementing a long short term memory (LSTM) cell, and wherein a last hidden state of the LSTM cell is selected as the HOI representation.

13. A computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method for improving video understanding tasks based on higher-order object interactions (HOIs) between object features comprising:

obtaining a plurality of frames of a video;

generating a coarse-grained feature representation, including generating an image feature for each of a plurality of timesteps respectively corresponding to each of the frames, and performing attention based on the image features;

generating a fine-grained feature representation, including generating an object feature for each of the plurality of timesteps, and generating the HOIs between the object features; and

concatenating the coarse-grained and the fine-grained feature representations to generate a concatenated feature representation.

14. The computer program product of claim 13 , wherein the method further comprises making a prediction based on the concatenated feature representation to perform one or more operations, and wherein the one or more operations include at least one of action recognition and video captioning.

15. The computer program product of claim 13 , wherein the image features are generated using a convolutional neural network (CNN).

16. The computer program product of claim 13 , wherein performing attention based on the image features further includes performing Scaled Dot Product (SDP) attention.

17. The computer program product of claim 13 , wherein the coarse-grained feature representation includes attended context information, and wherein generating the coarse-grained feature representation further comprises generating the attended context information by mean-pooling weighted image representations.

18. The computer program product of claim 13 , wherein generating the HOIs further includes:

generating projected object features respectively corresponding to the object features;

computing attention weights based at least in part on the projected object features;

performing attentive selection using a plurality of attentive selection components to generate attended object features each corresponding to a respective one of the attentive selection components;

concatenating the attended object features to generate an attended object feature representation; and

determining an HOI representation based on the attended object feature representation.

19. The computer program product of claim 18 , wherein determining the HOI representation further includes implementing a long short term memory (LSTM) cell, and wherein a last hidden state of the LSTM cell is selected as the HOI representation.

20. A system for improving video understanding tasks based on higher-order object interactions (HOIs) between object features, comprising:

at least one processor operatively coupled to a memory, wherein the at least one processor is configured to:

obtain a plurality of frames of a video;

generate a coarse-grained feature representation by generating an image feature for each of a plurality of timesteps respectively corresponding to each of the frames, and performing attention based on the image features;

generate a fine-grained feature representation by generating an object feature for each of the plurality of timesteps, and generating the HOIs between the object features; and

concatenate the coarse-grained and the fine-grained feature representations to generate a concatenated feature representation.