Deep 3D attention long short-term memory for video-based action recognition

1. A computer-implemented method for video based action recognition, the method comprising:

receiving, by a processor, one or more frames from one or more video sequences;

generating, by the processor using a deep convolutional neural network, a feature vector for each patch of the one or more frames;

updating the feature vectors with a domain knowledge infusion of a target action class corresponding to a target action by dot product;

generating, by the processor, an attention factor for the feature vectors based on a within-frame attention and a between-frame attention;

identifying, by the processor, the target action using a multi-layer deep long short-term memory process applied to the attention factor, said target action representing at least one of the one or more video sequences; and

controlling an operation of a processor-based machine to change a state of the processor-based machine, responsive to the at least one of one or more video sequences including the identified target action.

2. The computer-implemented method of claim 1 , wherein the within-frame attention and the between-frame attention use at least one of a softmax layer and a bidirectional long short-term memory process.

3. The computer-implemented method of claim 1 , wherein the within-frame attention and the between-frame attention include an attention selected from the group consisting of a hard attention and a soft attention.

4. The computer-implemented method of claim 1 , wherein the target action is selected from a group of action classes.

5. The computer-implemented method of claim 1 , wherein the long short-term memory process includes both hidden states and cell states.

6. The computer-implemented method of claim 5 , wherein the hidden states are in a hidden state dimension.

7. The computer-implemented method of claim 1 , wherein the multi-layer deep long short-term memory process utilizes a cross-entropy loss function.

8. The computer-implemented method of claim 7 , wherein the cross-entropy loss function includes a function selected from the group consisting of a last time point cross-entropy loss function, an all-time point cross-entropy loss function, and a max-neighbor cross-entropy loss function.

9. The computer-implemented method of claim 1 , wherein the within-frame attention includes a multilayer perceptron feeding into a softmax layer.

10. A computer program product for video based action recognition, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:

receiving, by a processor, one or more frames from one or more video sequences;

generating, by the processor using a deep convolutional neural network, a feature vector for each patch of the one or more frames;

updating the feature vectors with a domain knowledge infusion of a target action class corresponding to a target action by dot product;

generating, by the processor, an attention factor for the feature vectors based on a within-frame attention and a between-frame attention;

identifying, by the processor, the target action using a multi-layer deep long short-term memory process applied to the attention factor, said target action representing at least one of the one or more video sequences; and

controlling an operation of a processor-based machine to change a state of the processor-based machine, responsive to the at least one of the one or more video sequences including the identified target action.

11. The computer program product of claim 10 , wherein the within-frame attention includes a multilayer perceptron feeding into a softmax layer.

12. The computer program product of claim 10 , wherein the within-frame attention and the between-frame attention include an attention selected from the group consisting of a hard attention and a soft attention.

13. The computer program product of claim 10 , wherein the target action is selected from a group of action classes.

14. The computer program product of claim 10 , wherein the within-frame attention and the between-frame attention use at least one of a softmax layer and a bidirectional long short-term memory process.

15. The computer program product of claim 10 , wherein the long short-term memory process includes both hidden states and cell states.

16. The computer program product of claim 15 , wherein the hidden states are in a hidden state dimension.

17. The computer program product of claim 10 , wherein the within-frame attention and the between-frame attention use at least one of a softmax layer and a bidirectional long short-term memory process.

18. A system for video based action recognition, the system comprising:

a processor, configured to:

receive one or more frames from one or more video sequences;

generate, using a deep convolutional neural network, a feature vector for each patch of the one or more frames;

update the feature vectors with a domain knowledge infusion of a target action class corresponding to a target action by dot product;

generate an attention factor for the feature vectors based on a within-frame attention and a between-frame attention;

identify the target action using a multi-layer deep long short-term memory process applied to the attention factor, said target action representing at least one of the one or more video sequences; and

control an operation of a processor-based machine to change a state of the processor-based machine, responsive to the at least one of the one or more video sequences including the identified target action.