Video analysis with convolutional attention recurrent neural networks
A method of processing data within a convolutional attention recurrent neural network (RNN) includes generating a current multi-dimensional attention map. The current multi-dimensional attention map indicates areas of interest in a first frame from a sequence of spatio-temporal data. The method further includes receiving a multi-dimensional feature map. The method also includes convolving the current multi-dimensional attention map and the multi-dimensional feature map to obtain a multi-dimensional hidden state and a next multi-dimensional attention map. The method identifies a class of interest in the first frame based on the multi-dimensional hidden state and training data.
1. A method of processing data within a convolutional attention recurrent neural network (RNN), comprising:
generating a current multi-dimensional attention map, the current multi-dimensional attention map indicating areas of interest in a first frame from a sequence of spatio-temporal data;
receiving a multi-dimensional feature map;
convolving the current multi-dimensional attention map and the multi-dimensional feature map to obtain a multi-dimensional hidden state and a next multi-dimensional attention map; and
identifying a class of interest in the first frame based on the multi-dimensional hidden state and training data.
2. The method of claim 1 , further comprising determining a class of interest in the sequence of spatio-temporal data based on the class of interest in the first frame and a class of interest of at least a second frame from the sequence of spatio-temporal data.
3. The method of claim 1 , in which the class of interest is at least one of an action, an object, an event, or a combination thereof.
4. The method of claim 1 , in which the convolving comprises applying a multi-dimensional filter.
5. The method of claim 4 , in which the convolving is further based on a prior multi-dimensional hidden state.
6. The method of claim 1 , in which the convolutional attention RNN comprises an attention long short-term memory (LSTM) network.
7. The method of claim 1 , in which the multi-dimensional feature map is based on the first frame.
8. The method of claim 1 , in which, after a first time step, the current multi-dimensional attention map is based on a prior multi-dimensional attention map and a second frame of the sequence of spatio-temporal data.
9. The method claim 1 , further comprising receiving the multi-dimensional feature map from a layer of a convolutional neural network (CNN).
10. The method of claim 1 , in which the spatio-temporal data is a video.
11. An apparatus for processing data within a convolutional attention recurrent neural network (RNN), comprising:
a memory; and
at least one processor coupled to the memory, the at least one processor configured:
to generate a current multi-dimensional attention map, the current multi-dimensional attention map indicating areas of interest in a first frame from a sequence of spatio-temporal data;
to receive a multi-dimensional feature map;
to convolve the current multi-dimensional attention map and the multi-dimensional feature map to obtain a multi-dimensional hidden state and a next multi-dimensional attention map; and
to identify a class of interest in the first frame based on the multi-dimensional hidden state and training data.
12. The apparatus of claim 11 , in which the at least one processor is further configured to determine a class of interest in the sequence of spatio-temporal data based on the class of interest in the first frame and a class of interest of at least a second frame from the sequence of spatio-temporal data.
13. The apparatus of claim 11 , in which the class of interest is at least one of an action, an object, an event, or a combination thereof.
14. The apparatus of claim 11 , in which the at least one processor is further configured to apply a multi-dimensional filter to convolve the current multi-dimensional attention map and the multi-dimensional feature map.
15. The apparatus of claim 14 , in which the at least one processor is further configured to convolve based on a prior multi-dimensional hidden state.
16. The apparatus of claim 11 , in which the convolutional attention RNN comprises an attention long short-term memory (LSTM) network.
17. The apparatus of claim 11 , in which the multi-dimensional feature map is based on the first frame.
18. The apparatus of claim 11 , in which, after a first time step, the current multi-dimensional attention map is based on a prior multi-dimensional attention map and a second frame of the sequence of spatio-temporal data.
19. The apparatus claim 11 , in which the at least one processor is further configured to receive the multi-dimensional feature map from a layer of a convolutional neural network (CNN).
20. The apparatus of claim 11 , in which the spatio-temporal data is a video.
21. A non-transitory computer-readable medium having program code recorded thereon for processing data within a convolutional attention recurrent neural network (RNN), the program code being executed by a processor and comprising:
program code to generate a current multi-dimensional attention map, the current multi-dimensional attention map indicating areas of interest in a first frame from a sequence of spatio-temporal data;
program code to receive a multi-dimensional feature map;
program code to convolve the current multi-dimensional attention map and the multi-dimensional feature map to obtain a multi-dimensional hidden state and a next multi-dimensional attention map; and
program code to identify a class of interest in the first frame based on the multi-dimensional hidden state and training data.
22. The non-transitory computer-readable medium of claim 21 , in which program code further comprises program code to determine a class of interest in the sequence of spatio-temporal data based on the class of interest in the first frame and a class of interest of at least a second frame from the sequence of spatio-temporal data.
23. The non-transitory computer-readable medium of claim 21 , in which the class of interest is at least one of an action, an object, an event, or a combination thereof.
24. The non-transitory computer-readable medium of claim 21 , in which the program code further comprises program code to apply a multi-dimensional filter to convolve the current multi-dimensional attention map and the multi-dimensional feature map.
25. The non-transitory computer-readable medium of claim 24 , in which the program code to apply further comprises program code to convolve based on a prior multi-dimensional hidden state.
26. An apparatus for processing data within a convolutional attention recurrent neural network (RNN), comprising:
means for generating a current multi-dimensional attention map, the current multi-dimensional attention map indicating areas of interest in a first frame from a sequence of spatio-temporal data;
means for receiving a multi-dimensional feature map;
means for convolving the current multi-dimensional attention map and the multi-dimensional feature map to obtain a multi-dimensional hidden state and a next multi-dimensional attention map; and
means for identifying a class of interest in the first frame based on the multi-dimensional hidden state and training data.
27. The apparatus of claim 26 , further comprising means for determining a class of interest in the sequence of spatio-temporal data based on the class of interest in the first frame and a class of interest of at least a second frame from the sequence of spatio-temporal data.
28. The apparatus of claim 26 , in which the class of interest is at least one of an action, an object, an event, or a combination thereof.
29. The apparatus of claim 26 , in which the means for convolving further comprises means for applying a multi-dimensional filter.
30. The apparatus of claim 29 , in which the means for convolving is further based on a prior multi-dimensional hidden state.