Self-attention deep neural network for action recognition in surveillance videos
An artificial neural network for analyzing input data, the input data being a 3D tensor having D channels, such as D frames of a video snippet, to recognize an action therein, including: D spatial transformer modules, each generating first and second spatial transformations and corresponding first and second attention windows using only one of the D channels, and transforming first and second regions of each of the D channels corresponding to the first and second attention windows to generate first and second patch sequences; first and second CNNs, respectively processing a concatenation of the D first patch sequences and a concatenation of the D second patch sequences; and a classification network receiving a concatenation of the outputs of the first and second CNNs and the D sets of transformation parameters of the first transformation outputted by the D spatial transformer modules, to generate a predicted action class.
1. An artificial neural network implemented on a data processing system for analyzing input data to recognize an action therein, the input data being a 3-dimensional input tensor having D channels, D being a natural number greater than 1, the artificial neural network comprising:
D spatial transformer neural network modules,
wherein each spatial transformer module is configured to receive the input tensor, generate a first and a second spatial transformation using only one of the D channels, generate a first and a second attention window respectively corresponding to the first and the second spatial transformation, transform a first and a second region of each of the D channels of the input tensor corresponding to the first and the second attention window to a first and a second output patch for that channel, the first and the second output patches for all D channels respectively forming a first and a second patch sequence,
wherein each spatial transformer module is further configured to output a set of first transformation parameters of the first spatial transformation,
wherein different ones of the D spatial transformer modules are configured to use different ones of the D channels of the input tensor to generate their first and second spatial transformations;
a first and a second convolutional neural network (CNN), the first CNN being configured to receive as input a concatenation of the D first patch sequences generated by the D spatial transformer modules and to generate a first CNN output, the second CNN being configured to receive as input a concatenation of the D second patch sequences generated by the D spatial transformer modules and to generate a second CNN output; and
a classification neural network, configured to receive as input a concatenation of the first CNN output, the second CNN output, and the D sets of first transformation parameters outputted by the D spatial transformer modules, and to generate a predicted action class.
2. The artificial neural network of claim 1 , wherein the input data is a video snippet including a time sequence of D frames, each frame being a channel of the input tensor.
3. The artificial neural network of claim 1 , wherein each spatial transformer neural network module includes:
a localization neural network, configured to receive the one of the D channels of the input tensor and to generate the set of first transformation parameters and a set of second transformation parameters;
a first grid generator, configured to generate a first sampling grid by transforming a first regular grid according to the set of first transformation parameters into the first sampling grid over the one channel of the input tensor;
a second grid generator, configured to generate a second sampling grid by transforming a second regular grid according to the set of second transformation parameters into the second sampling grid over the one channel of the input tensor; and
a sampler, configured to apply the first sampling grid to sample each one of the D channels of the input tensor to generate the first patch sequence, and to apply the second sampling grid to sample each one of the D channels of the input tensor to generate the second patch sequence.
4. The artificial neural network of claim 1 , wherein the classification neural network includes a fully connected layer and a softmax layer.
5. The artificial neural network of claim 1 , wherein each of the first and second spatial transformations includes one or more of cropping, translation, rotation, scaling, skew, projective transformation, and elastic warping.
6. A method performed by an artificial neural network implemented on a data processing system, to analyze input data to recognize an action therein, the input data being a 3-dimensional input tensor having D channels, D being a natural number greater than 1, the artificial neural network comprising D spatial transformer neural network modules, a first and a second convolutional neural network (CNN), and a classification neural network, the method comprising:
by each of the D spatial transformer neural network modules:
receiving the input tensor;
using only one of the D channels of the input tensor, generating a set of first transformation parameters and a set of second transformation parameters;
generating a first sampling grid by transforming a first regular grid according to the set of first transformation parameters into the first sampling grid over the one channel of the input tensor;
generating a second sampling grid by transforming a second regular grid according to the set of second transformation parameters into the second sampling grid over the one channel of the input tensor;
applying the first sampling grid to sample each one of the D channels of the input tensor to generate a first patch sequence; and
applying the second sampling grid to sample each one of the D channels of the input tensor to generate a second patch sequence;
wherein different ones of the D spatial transformer modules use different ones of the D channels of the input tensor to generate their first and second spatial transformations;
concatenating the D first patch sequences generated by the D spatial transformer modules into first concatenated data;
by the first CNN, receiving the first concatenated data as input and generating a first CNN output;
concatenating the D second patch sequences generated by the D spatial transformer modules into second concatenated data;
by the second CNN, receiving the second concatenated data as input and generating a second CNN output;
concatenating the first CNN output, the second CNN output, and the D sets of first transformation parameters generated by the D spatial transformer modules into a final feature vector; and
by the classification neural network, receiving the final feature vector as input and generating a predicted action class.
7. The method of claim 6 , wherein the input data is a video snippet including a time sequence of D frames, each frame being a channel of the input tensor.
8. The method of claim 6 , wherein each of the sets of first and second transformation parameters describes a transformation that includes one or more of cropping, translation, rotation, scaling, skew, projective transformation, and elastic warping.