SKIP CONVOLUTIONS FOR EFFICIENT VIDEO PROCESSING
A method for video processing via an artificial neural network includes receiving a video stream as an input at the artificial neural network. A residual is computed based on a difference between a first feature of a current frame of the video stream and a second feature of a previous frame of the video stream. One or more portions of the current frame of the video stream are processed based on the residual. Additionally, processing is skipped for one or more portions of the current frame of the video based on the residual.
1 . A method for video processing with an artificial neural network (ANN), comprising:
receiving a video stream as an input at the artificial neural network;
computing a residual based on a difference between a first feature of a current frame of the video stream and a second feature of a previous frame of the video stream; and
determining to perform processing, associated with a convolutional layer of the artificial neural network, of one or more portions of the current frame of the video stream based on of the residual.
2 . The method of claim 1 , in which the one or more portions of the current frame includes only salient regions of the current frame.
3 . The method of claim 2 , further comprising applying, based on the determination to perform processing, a convolution kernel to only the salient regions of the current frame.
4 . The method of claim 2 , further comprising determining the one or more salient regions when the residual is greater than a predetermined threshold value.
5 . The method of claim 1 , in which a first output corresponding to the one or more portions of the current frame is set equal to a second output corresponding to at least one portion of the previous frame.
6 . The method of claim 1 , further comprising:
comparing the residual to a predefined threshold value; and
determining to apply a mask to the first feature based on the comparing.
7 . The method of claim 1 , further comprising learning a gating function for gating the convolutional layer, the gating function configured to apply a mask to the one or more portions of the current frame based on the residual.
8 . The method of claim 7 , further comprising generating a saliency map based on the gating function.
9 . The method of claim 1 , further comprising adaptively adjusting an amount of computation performed in processing the video stream based on an amount of information observed per frame.
10 . An apparatus for video processing with an artificial neural network (ANN), comprising:
at least one memory; and
at least one processor coupled to the at least one memory, the at least one processor being configured to:
receive a video stream as an input at the artificial neural network;
compute a residual based on a difference between a first feature of a current frame of the video stream and a second feature of a previous frame of the video stream; and
determine to perform processing, associated with a convolutional layer of the artificial neural network, of one or more portions of the current frame of the video stream based on of the residual.
11 . The apparatus of claim 10 , in which the one or more portions of the current frame includes only salient regions of the current frame.
12 . The apparatus of claim 11 , in which the at least one processor is further configured to determine the one or more salient regions when the residual is greater than a predetermined threshold value.
13 . The apparatus of claim 10 , in which a first output corresponding to the one or more portions of the current frame is set equal to a second output corresponding to at least one portion of the previous frame.
14 . The apparatus of claim 10 , in which the at least one processor is further configured to:
compare the residual to a predefined threshold value; and
determine to apply a mask to the first feature based on the comparing.
15 . The apparatus of claim 10 , in which the at least one processor is further configured to learn a gating function for gating the convolutional layer, the gating function configured to apply a mask to the one or more portions of the current frame based on the residual.
16 . The apparatus of claim 15 , in which the at least one processor is further configured to generate a saliency map based on the gating function.
17 . The apparatus of claim 10 , in which the at least one processor is further configured to adaptively adjust an amount of computation based on an amount of information observed per frame.
18 . A non-transitory computer-readable medium having encoded thereon program code for video processing with an artificial neural network (ANN), the program code being executed by a processor and comprising:
program code to receive a video stream as an input at the artificial neural network;
program code to compute a residual based on a difference between a first feature of a current frame of the video stream and a second feature of a previous frame of the video stream; and
program code to determine to perform processing, associated with a convolutional layer of the artificial neural network, of one or more portions of the current frame of the video stream based on of the residual.
19 . The non-transitory computer-readable medium of claim 18 , in which the one or more portions of the current frame includes only salient regions of the current frame.
20 . The non-transitory computer-readable medium of claim 19 , in which the at least one processor is further configured to determine the one or more salient regions when the residual is greater than a predetermined threshold value.