4D convolutional neural networks for video recognition
This disclosure includes technologies for video recognition in general. The disclosed system can automatically detect various types of actions in a video, including reportable actions that cause shrinkage in a practical application for loss prevention in the retail industry. The temporal evolution of spatio-temporal features in the video are used for action recognition. Such features may be learned via a 4D convolutional operation, which is adapted to model low-level features based on a residual 4D block. Further, appropriate responses may be invoked if a reportable action is recognized.
1. A computer-implemented method for video recognition, comprising:
modeling, based on a residual four-dimensional (4D) block in a neural network, both short-range temporal structural features and long-range temporal structural features in a video;
preserving three-dimensional (3D) spatio-temporal representations between two 3D convolutional layers in the neural network with a residual connection in the residual 4D block;
recognizing, based on the short-range temporal structural features and the long-range temporal structural features, an action in the video; and
causing display of a message in response to the action being recognized.
2. The method of claim 1 , wherein the long-range temporal structural features comprises evolution features of respective 3D features of a plurality of action units.
3. The method of claim 2 , wherein the short-range temporal structural features comprises 3D features of an action unit, the action unit including a plurality of frames selected from the video.
4. The method of claim 3 , further comprising: aggregating, via the residual 4D block, 3D features of the action unit and the evolution features of respective 3D features of the plurality of action units, for video-level action recognition.
5. The method of claim 3 , further comprising: modeling the evolution features of respective 3D features of the plurality of action units via a plurality of concatenated 4D convolution blocks, one 4D convolution block including a plurality of 3D convolutional layers followed by the residual 4D block.
6. The method of claim 1 , further comprising:
forming an input to the residual 4D block as a tensor with a first dimension of a channel, a second dimension of an action unit, a third dimension of a temporal length of the action unit, a fourth dimension of a height of the action unit, and a fifth dimension of a width of the action unit, wherein the action unit is a selected segment of the video.
7. The method of claim 1 , further comprising: implementing a 4D convolutional operation with the residual 4D block based on a summation, in a dimension of action unit, of a plurality of action-unit-based 3D convolutional operations that use a temporal length, a height, and a width of an action unit at a channel.
8. The method of claim 1 , further comprising:
classifying, based on the action, the video into a reportable type; and
generating the message in response to the video being classified into the reportable type.
9. A non-transitory computer-readable storage device encoded with instructions that, when executed, cause one or more processors of a computing system to perform operations of video recognition, comprising:
modeling, based on a residual four-dimensional (4D) block in a neural network, both short-range temporal structural features and long-range temporal structural features in a video;
preserving three-dimensional (3D) spatio-temporal representations between two 3D convolutional layers in the neural network with a residual connection in the residual 4D block;
recognizing, based on the short-range temporal structural features and the long-range temporal structural features, an action in the video; and
causing display of a message in response to the action being recognized.
10. The computer-readable storage device of claim 9 , wherein the long-range temporal structural features comprises evolution features of respective 3D features of a plurality of action units.
11. The computer-readable storage device of claim 10 , wherein the short-range temporal structural features comprises 3D features of an action unit, the action unit including a plurality of frames selected from the video.
12. The computer-readable storage device of claim 11 , wherein the instructions that, when executed, cause the one or more processors to perform further operations comprising:
aggregating, via the residual 4D block, 3D features of the action unit and the evolution features of respective 3D features of the plurality of action units, for video-level action recognition.
13. The computer-readable storage device of claim 11 , wherein the instructions that, when executed, cause the one or more processors to perform further operations comprising:
modeling the evolution features of respective 3D features of the plurality of action units via a plurality of concatenated 4D convolution blocks, one 4D convolution block including a plurality of 3D convolutional layers followed by the residual 4D block.
14. The computer-readable storage device of claim 9 , wherein the instructions that, when executed, cause the one or more processors to perform further operations comprising:
forming an input to the residual 4D block as a tensor with a first dimension of a channel, a second dimension of an action unit, a third dimension of a temporal length of the action unit, a fourth dimension of a height of the action unit, and a fifth dimension of a width of the action unit, wherein the action unit is a selected segment of the video.
15. The computer-readable storage device of claim 9 , wherein the instructions that, when executed, further cause the one or more processors to perform operations comprising:
implementing a 4D convolutional operation with the residual 4D block based on a summation, in a dimension of action unit, of a plurality of action-unit-based 3D convolutional operations that use a temporal length, a height, and a width of an action unit at a channel.
16. A system for RGB-based video recognition, comprising: a processor; and
a memory having instructions stored thereon, wherein the instructions, when executed by the processor, cause the processor to:
identify, via an action localization operation, a section of a video corresponding to an action;
determine, via a four-dimensional (4D) convolutional operation in a neural network, a temporal evolution of spatio-temporal features in the section of the video;
classify, based on the temporal evolution of spatio-temporal features, the action as a type of reportable action, wherein the type of reportable action comprises a type of theft;
cause display of a message based on the action being classified as the type of reportable action;
extract the section of the video from the video; and
wirelessly transmit the section of the video in the message to a remote device, wherein the message comprises a description of the type of theft.
17. The system of claim 16 , wherein the instructions, when executed by the processor, further cause the processor to:
jointly capture local and global three-dimensional (3D) spatio-temporal features of the section of the video based on the 4D convolutional operation.
18. The system of claim 16 , wherein the instructions, when executed by the processor, further cause the processor to:
implement the 4D convolutional operation based on a summation, in a dimension of action unit, of a plurality of action-unit-based 3D convolutional operations, wherein an action unit represents a random segment of the section of the video.
19. The system of claim 16 , wherein the instructions, when executed by the processor, further cause the processor to:
generate the message.
20. The system of claim 16 , wherein the type of reportable action comprises a type of theft, and the message comprises a description of the type of theft, wherein the instructions, when executed by the processor, further cause the processor to:
extract the section of the video from the video; and
wirelessly transmit the section of the video in the message to a remote device.