Dynamic temporal normalization for deep learning in video understanding applications
Techniques related to application of deep neural networks to video for video recognition and understanding are discussed. A feature map of a deep neural network for a current time stamp of input video is standardized to a standardized feature map and pooled to a feature vector. The feature vector and transform parameters for a prior time stamp are used to generate transform parameters for the current time stamp based on application of a meta temporal relay. The resultant current time stamp transform parameters, such as a hidden state and a cell state of the meta temporal relay, are used to transform the standardized feature map to a normalized feature map for use by a subsequent layer of the deep neural network.
1 . A system for application of video recognition to input video, comprising:
a memory to store at least a portion of a current standardized feature map from a current layer of a deep neural network, the current standardized feature map corresponding to a current time stamp of the input video; and
one or more processors coupled to the memory, the one or more processors to:
generate current transform parameters for linear transformation of the current standardized feature map based on prior transform parameters and a current feature map corresponding to the current standardized feature map, the prior transform parameters used to transform a prior standardized feature map to a prior normalized feature map for a prior time stamp of the input video;
transform the current standardized feature map to a current normalized feature map using the current transform parameters, wherein the current transform parameters include a hidden state parameter and a cell state parameter of a meta temporal relay applied to the prior transform parameters and the current feature map; and
generate a video recognition output based on application of a subsequent layer of the deep neural network to the current normalized feature map, wherein the subsequent layer of the deep neural network is subsequent to the current layer of the deep neural network.
2 . The system of claim 1 , wherein the one or more processors to transform the current standardized feature map to the current normalized feature map comprises the one or more processors to sum the cell state parameter with a product of a first standardized feature of the current standardized feature map and the hidden state parameter.
3 . The system of claim 1 , wherein the meta temporal relay comprises a mapping from a current feature vector and a prior hidden state to one or more gating mechanisms applied by the meta temporal relay, the prior hidden state used to transform the prior standardized feature map to the prior normalized feature map for the prior time stamp of the input video.
4 . The system of claim 3 , wherein the mapping comprises a bottleneck mapping structure from the current feature vector and the prior hidden state to the one or more gating mechanisms.
5 . The system of claim 1 , the one or more processors to:
generate the current standardized feature map by standardizing the current feature map based on a mean and a standard deviation estimated using at least a portion of the input video or the current feature map.
6 . The system of claim 5 , wherein the one or more processors to generate the current transform parameters is based on a current feature vector, the one or more processors to: generate the current feature vector by applying global average pooling to the current feature map.
7 . The system of claim 5 , wherein, in the linear transformation of the current standardized feature map to the current normalized feature map, the current transform parameters are used in place of the mean and the standard deviation.
8 . The system of claim 1 , wherein the deep neural network comprises one of a video object detection model, a video action detection model, or a video segmentation model.
9 . The system of claim 1 , wherein the current layer of the deep neural network is an input layer of the deep neural network.
10 . A method for application of video recognition to input video, comprising:
receiving a current standardized feature map from a current layer of a deep neural network, the current standardized feature map corresponding to a current time stamp of the input video;
generating current transform parameters for linear transformation of the current standardized feature map based on prior transform parameters and a current feature map corresponding to the current standardized feature map, the prior transform parameters used to transform a prior standardized feature map to a prior normalized feature map for a prior time stamp of the input video;
transforming the current standardized feature map to a current normalized feature map using the current transform parameters, wherein the current transform parameters include a hidden state parameter and a cell state parameter of a meta temporal relay applied to the prior transform parameters and the current feature map; and
generating a video recognition output based on application of a subsequent layer of the deep neural network to the current normalized feature map, wherein the subsequent layer of the deep neural network is subsequent to the current layer of the deep neural network.
11 . The method of claim 10 , wherein transforming the current standardized feature map to the current normalized feature map comprises summing the cell state parameter with a product of a first standardized feature of the current standardized feature map and the hidden state parameter.
12 . The method of claim 10 , wherein the meta temporal relay comprises a mapping from a current feature vector and a prior hidden state to one or more gating mechanisms applied by the meta temporal relay, the prior hidden state used to transform the prior standardized feature map to the prior normalized feature map for the prior time stamp of the input video.
13 . The method of claim 12 , wherein the mapping comprises a bottleneck mapping structure from the current feature vector and the prior hidden state to the one or more gating mechanisms.
14 . At least one non-transitory machine readable medium comprising machine-readable instructions that, in response to being executed on a device, cause the device to apply video recognition to input video by:
receiving a current standardized feature map from a current layer of a deep neural network, the current standardized feature map corresponding to a current time stamp of the input video;
generating current transform parameters for linear transformation of the current standardized feature map based on prior transform parameters and a current feature map corresponding to the current standardized feature map, the prior transform parameters used to transform a prior standardized feature map to a prior normalized feature map for a prior time stamp of the input video;
transforming the current standardized feature map to a current normalized feature map using the current transform parameters, wherein the current transform parameters include a hidden state parameter and a cell state parameter of a meta temporal relay applied to the prior transform parameters and the current feature map; and
generating a video recognition output based on application of a subsequent layer of the deep neural network to the current normalized feature map, wherein the subsequent layer of the deep neural network is subsequent to the current layer of the deep neural network.
15 . The at least one non-transitory machine readable medium of claim 14 , wherein transforming the current standardized feature map to the current normalized feature map comprises summing the cell state parameter with a product of a first standardized feature of the current standardized feature map and the hidden state parameter.
16 . The at least one non-transitory machine readable medium of claim 14 , wherein the meta temporal relay comprises a mapping from a current feature vector and a prior hidden state to one or more gating mechanisms applied by the meta temporal relay, the prior hidden state used to transform the prior standardized feature map to the prior normalized feature map for the prior time stamp of the input video.
17 . The at least one non-transitory machine readable medium of claim 16 , wherein the mapping comprises a bottleneck mapping structure from the current feature vector and the prior hidden state to the one or more gating mechanisms.