Method and system for video action classification by mixing 2D and 3D features

1. A computer-implemented method for classifying video action, the method comprising:

receiving, by an information handling system comprising a processor and a memory, a video for action analysis;

loading, by the information handling system, a plurality of processing pathways comprising a 2D convolutional neural network processing pathway and a 3D convolutional neural network processing pathway that is formed by inflating the 2D convolutional neural network processing pathway;

selecting, by the information handling system, a first video frame and a first plurality of video frames from the video;

processing, by the information handling system, the first video frame with the 2D convolutional neural network processing pathway to extract spatial features classifying the first video frame;

processing, by the information handling system, the first plurality of video frames with the 3D convolutional neural network processing pathway to extract spatiotemporal features classifying the first plurality of video frames; and

combining, by the information handling system, the spatial features with the spatiotemporal features to generate a classification label for the video action.

2. The computer-implemented method of claim 1 , further comprising initializing, by the information handling system, the 2D convolutional neural network processing pathway and the 3D convolutional neural network processing pathway with pretrained weights.

3. The computer-implemented method of claim 2 , further comprising training, by the information handling system, a final fully connected layer of the 2D convolutional neural network processing pathway and the 3D convolutional neural network processing pathway with the pretrained weights for a first specified epoch training period.

4. The computer-implemented method of claim 3 , further comprising:

training, after the first specified epoch training period, all layers of the 2D convolutional neural network processing pathway and the 3D convolutional neural network processing pathway with the pretrained weights for a second specified epoch training period.

5. The computer-implemented method of claim 1 , where the spatial features at each stage of the 2D convolutional neural network processing pathway are fused with corresponding spatiotemporal features at each stage of the 3D convolutional neural network processing pathway.

6. The computer-implemented method of claim 1 , where combining the spatial features with the spatiotemporal features comprises concatenating a final spatial feature from the 2D convolutional neural network processing pathway with a final spatiotemporal feature from the 3D convolutional neural network processing pathway to connect to a final, fully connected layer that is used for classifying the video action of the video.

7. An information handling system comprising:

one or more processors;

a memory coupled to at least one of the processors;

a set of instructions stored in the memory and executed by at least one of the processors to classify video action, wherein the set of instructions are executable to perform actions of:

receiving, by the system, a video for action analysis;

loading, by the system, a plurality of processing pathways comprising a 2D convolutional neural network processing pathway and a 3D convolutional neural network processing pathway that is formed by inflating the 2D convolutional neural network processing pathway;

selecting, by the system, a first video frame and a first plurality of video frames from the video;

processing, by the system, the first video frame with the 2D convolutional neural network processing pathway to extract spatial features classifying the first video frame;

processing, by the system, the first plurality of video frames with the 3D convolutional neural network processing pathway to extract spatiotemporal features classifying the first plurality of video frames; and

combining, by the system, the spatial features with the spatiotemporal features to generate a classification label for the video action.

8. The information handling system of claim 7 , wherein the set of instructions are executable to initialize the 2D convolutional neural network processing pathway and the 3D convolutional neural network processing pathway with pretrained weights.

9. The information handling system of claim 8 , wherein the set of instructions are executable to train a final fully connected layer of the 2D convolutional neural network processing pathway and the 3D convolutional neural network processing pathway with the pretrained weights for a first specified epoch training period.

10. The information handling system of claim 9 , wherein the set of instructions are executable to train, after the first specified epoch training period, all layers of the 2D convolutional neural network processing pathway and the 3D convolutional neural network processing pathway with the pretrained weights for a second specified epoch training period.

11. The information handling system of claim 7 , where the spatial features at each stage of the 2D convolutional neural network processing pathway are fused with corresponding spatiotemporal features at each stage of the 3D convolutional neural network processing pathway.

12. The information handling system of claim 7 , wherein the set of instructions are executable to provide combine the spatial features with the spatiotemporal features by concatenating a final spatial feature from the 2D convolutional neural network processing pathway with a final spatiotemporal feature from the 3D convolutional neural network processing pathway to connect to a final, fully connected layer that is used for classifying the video action of the video.

13. A computer program product stored in a computer readable storage medium, comprising computer instructions that, when executed by an information handling system comprising a processor and a memory, causes the system to classify video action by:

receiving, by the system, a video for action analysis;

loading, by the system, a plurality of processing pathways comprising a 2D convolutional neural network processing pathway and a 3D convolutional neural network processing pathway that is formed by inflating the 2D convolutional neural network processing pathway;

selecting, by the system, a first video frame and a first plurality of video frames from the video;

processing, by the system, the first video frame with the 2D convolutional neural network processing pathway to extract spatial features classifying the first video frame;

processing, by the system, the first plurality of video frames with the 3D convolutional neural network processing pathway to extract spatiotemporal features classifying the first plurality of video frames; and

combining, by the system, the spatial features with the spatiotemporal features to generate a classification label for the video action.

14. The computer program product of claim 13 , further comprising computer instructions that, when executed by the system, causes the system to initialize the 2D convolutional neural network processing pathway and the 3D convolutional neural network processing pathway with pretrained weights.

15. The computer program product of claim 14 , further comprising computer instructions that, when executed by the system, causes the system to train a final fully connected layer of the 2D convolutional neural network processing pathway and the 3D convolutional neural network processing pathway with the pretrained weights for a first specified epoch training period.

16. The computer program product of claim 15 , further comprising computer instructions that, when executed by the system, causes the system to train, after the first specified epoch training period, all layers of the 2D convolutional neural network processing pathway and the 3D convolutional neural network processing pathway with the pretrained weights for a second specified epoch training period.

17. The computer program product of claim 13 , where the spatial features at each stage of the 2D convolutional neural network processing pathway are fused with corresponding spatiotemporal features at each stage of the 3D convolutional neural network processing pathway.