3D convolutional neural networks for automatic human action recognition
View Patent ↗Systems and methods are disclosed to recognize human action from one or more video frames by performing 3D convolutions to capture motion information encoded in multiple adjacent frames and extracting features from spatial and temporal dimensions therefrom; generating multiple channels of information from the video frames, combining information from all channels to obtain a feature representation for a 3D CNN model; and applying the 3D CNN model to recognize human actions.
1. A computer implemented method to automatically recognize human actions from one or more video frames, comprising:
performing 3D convolutions to capture motion information encoded in multiple adjacent frames and extracting features from spatial and temporal dimensions therefrom;
generating multiple channels of information from the video frames,
combining information from all channels to obtain a feature representation for a three dimensional convolution neural network (3D CNN) model including determining a value v at position (x, y, z) on a j th feature map in an i th layer of the 3D CNN as:
v
ij
xyz
=
tanh
(
b
ij
+
∑
m
∑
p
=
0
P
i
-
1
∑
q
=
0
Q
i
-
1
∑
r
=
0
R
i
-
1
w
ijm
pqr
v
(
i
-
1
)
m
(
x
+
p
)
(
y
+
q
)
(
z
+
r
)
)
,
where tanh(•) is the hyperbolic tangent function, P i and Q i are height and width of a 3D kernel, R i is the size of the 3D kernel along a temporal dimension, w ijm pqr is the (p, q, r) th value of the kernel connected to the m th feature map in a previous layer, and b ij is a feature map bias; and
applying the 3D CNN model to recognize human actions.
2. The method of claim 1 , comprising detecting humans in the video frames.
3. The method of claim 1 , comprising tracking humans in the video frames.
4. The method of claim 1 , comprising generating multi-frame cubes from tracked humans.
5. The method of claim 4 , wherein the multiple frames cubes are obtained by extracting bounding boxes at a predetermined position from consecutive frames before and after a current frame, leading to a cube containing an action.
6. The method of claim 1 , wherein the 3D CNN comprises a hardwired layer, convolution layers, subsampling layers, and a full connection layer.
7. The method of claim 6 , wherein the hardwired layer comprises kernels to generate the multiple channels of information from the video frame with feature maps in gray, horizontal gradient, vertical gradient, and optical flow gradients along horizontal and vertical directions, computed from adjacent video frames.
8. The method of claim 1 , comprising generating a feature vector capturing motion information in the input frames.
9. The method of claim 8 , comprising applying a linear classifier on the feature vector for action classification.
10. A system to automatically recognize human actions, comprising;
means for performing 3D convolutions to capture motion information encoded in multiple adjacent frames and extracting features from spatial and temporal dimensions therefrom;
means for generating multiple channels of information from the video frames,
means for combining information from all channels to obtain a feature representation for a three dimensional convolution neural network (3D CNN) model including determing a value v at position (x, y, z) on a j th feature map in an i th layer as:
v
ij
xyz
=
tanh
(
b
ij
+
∑
m
∑
p
=
0
P
i
-
1
∑
q
=
0
Q
i
-
1
∑
r
=
0
R
i
-
1
w
ijm
pqr
v
(
i
-
1
)
m
(
x
+
p
)
(
y
+
q
)
(
z
+
r
)
)
,
where tanh(•) is the hyperbolic tangent function, P i and Q i are height and width of a 3D kernel, R i is the size of the 3D kernel along a temporal dimension, w ijm pqr is the (p, q, r) th value of the kernel connected to the m th feature map in a previous layer, and b ij is a feature map bias; and
means for applying the 3D CNN model to recognize human actions.
11. The system of claim 10 , comprising a human detector coupled to the video frames.
12. The system of claim 10 , comprising a human tracker coupled to the video frames.
13. The system of claim 10 , comprising means for generating multi-frame cubes from tracked humans.
14. The system of claim 13 , wherein the multiple frames cubes are obtained by extracting bounding boxes at a predetermined position from consecutive frames before and after a current frame, leading to a cube containing an action.
15. The system of claim 10 , wherein the 3D CNN comprises a hardwired layer, convolution layers, subsampling layers, and a full connection layer.
16. The system of claim 15 , wherein the hardwired layer comprises kernels to generate the multiple channels of information from the video frame with feature maps in gray, horizontal gradient, vertical gradient, and optical flow gradients along horizontal and vertical directions, computed from adjacent video frames.
17. The system of claim 10 , comprising means for generating a feature vector capturing motion information in the input frames.
18. The system of claim 17 , comprising means for applying a linear classifier on the feature vector for action classification.