Spatio-temporal action and actor localization

1. A method of pixel-wise localization of an actor and an action in a sequence of frames, comprising:

generating a first set of filters based on a natural language query describing the action and the actor;

generating a visual representation for each frame of the sequence of frames;

generating a response map for each frame of the sequence of frames based on a convolution of the first set of filters and the visual representation of each frame;

generating a second set of dynamic filters based on the natural language query and the response map;

labeling pixels in each frame of the sequence of frames based on a convolution of the second set of dynamic filters and an up-sampled visual representation of each frame; and

localizing the action and the actor in the sequence of frames based on the labeled pixels.

2. The method of claim 1 , further comprising:

up-sampling a resolution of the visual representation;

and

convolving the up-sampled visual representation with the second set of dynamic filters.

3. The method of claim 2 , further comprising repeating the up-sampling and the convolving for a set of resolutions of the visual representation.

4. The method of claim 2 , in which the first set of filters and the second set of dynamic filters are two-dimensional or three-dimensional filters.

5. The method of claim 1 , further comprising controlling an apparatus based on the localized action and actor.

6. An apparatus for pixel-wise localization of an actor and an action in a sequence of frames, the apparatus comprising:

means for generating a first set of filters based on a natural language query describing the action and the actor;

means for generating a visual representation for each frame of the sequence of frames;

means for generating a response map for each frame of the sequence of frames based on a convolution of the first set of filters and the visual representation of each frame;

means for generating a second set of dynamic filters based on the natural language query and the response map;

means for labeling pixels in each frame of the sequence of frames based on a convolution of the second set of dynamic filters and an up-sampled visual representation of each frame; and

means for localizing the action and the actor in the sequence of frames based on the labeled pixels.

7. The apparatus of claim 6 , further comprising:

means for up-sampling a resolution of the visual representation;

and

means for convolving the up-sampled visual representation with the second set of dynamic filters.

8. The apparatus of claim 7 , further comprising means for repeating up-sampling and convolving for a set of resolutions of the visual representation.

9. The apparatus of claim 7 , in which the first set of filters and the second set of dynamic filters are two-dimensional or three-dimensional filters.

10. The apparatus of claim 6 , further comprising means for controlling the apparatus based on the localized action and actor.

11. An apparatus for pixel-wise localization of an actor and an action in a sequence of frames, the apparatus comprising:

a memory; and

at least one processor coupled to the memory, the at least one processor configured:

to generate a first set of filters based on a natural language query describing the action and the actor;

to generate a visual representation for each frame of the sequence of frames;

to generate a response map for each frame of the sequence of frames based on a convolution of the first set of filters and the visual representation of each frame;

to generate a second set of dynamic filters based on the natural language query and the response map;

to label pixels in each frame of the sequence of frames based on a convolution of the second set of dynamic filters and an up-sampled visual representation of each frame; and

to localize the action and the actor in the sequence of frames based on the labeled pixels.

12. The apparatus of claim 11 , in which the at least one processor is further configured to:

up-sample a resolution of the visual representation;

and

convolve the up-sampled visual representation with the second set of dynamic filters.

13. The apparatus of claim 12 , in which the at least one processor is further configured to up-sample and convolve for a set of resolutions of the visual representation.

14. The apparatus of claim 12 , in which the first set of filters and the second set of dynamic filters are two-dimensional or three-dimensional filters.

15. The apparatus of claim 11 , in which the at least one processor is further configured to control the apparatus based on the localized action and actor.

16. A non-transitory computer-readable medium having program code recorded thereon for pixel-wise localization of an actor and an action in a sequence of frames, the program code executed by a processor and comprising:

program code to generate a first set of filters based on a natural language query describing the action and the actor;

program code to generate a visual representation for each frame of the sequence of frames;

program code to generate a response map for each frame of the sequence of frames based on a convolution of the first set of filters and the visual representation of each frame;

program code to generate a second set of dynamic filters based on the natural language query and the response map;

program code to label pixels in each frame of the sequence of frames based on a convolution of the second set of dynamic filters and an up-sampled visual representation of each frame; and

program code to localize the action and the actor in the sequence of frames based on the labeled pixels.

17. The non-transitory computer-readable medium of claim 16 , in which the program code further comprises:

program code to up-sample a resolution of the visual representation; and

program code to convolve the up-sampled visual representation with the second set of dynamic filters.

18. The non-transitory computer-readable medium of claim 17 , in which the program code further comprises program code to up-sample and program code to convolve for a set of resolutions of the visual representation.

19. The non-transitory computer-readable medium of claim 17 , in which the first set of filters and the second set of dynamic filters are two-dimensional or three-dimensional filters.

20. The non-transitory computer-readable medium of claim 16 , in which the program code further comprises program code to control an apparatus based on the localized action and actor.