Low-power change-based neural network inference for image processing
One or more aspects of the present disclosure enable high accuracy computer vision and image processing techniques with decreased system resource requirements (e.g., with decreased computational load, shallower neural network designs, etc.). As described in more detail herein, one or more aspects of the described techniques may leverage key layers (e.g., certain key layers of a neural network) and compressed tensor comparisons to efficiently exploit temporal redundancy in videos and other slow changing signals (e.g., to efficiently reduce neural network inference computational burden, with only minor increase in data transfer power consumption). For example, key layers of a neural network may be identified, and temporal/spatial redundancy across frames may be efficiently leveraged such that only a computation region in a subsequent frame n+1 is re-computed in layers between identified key layers, while remaining feature-map calculations may be disabled in the layers between the identified key layers.
1. A method comprising:
receiving a video comprising a plurality of video frames including a first frame and a second frame;
identifying a set of key layers of a neural network including a first layer and a third layer and excluding a second layer between the first layer and the third layer;
computing a first projection matrix by performing a first convolution operation on the first frame at the first layer;
computing a second projection matrix for the first frame at the second layer;
computing a third projection matrix by performing the first convolution operation on the second frame at the first layer;
computing a projection mask indicating a computation region for the second frame based on the first projection matrix, the third projection matrix, and the first layer being included in the set of key layers;
disabling a first calculation and performing a second calculation at the second layer based on the third projection matrix, the projection mask, and the second layer being excluded from the set of key layers;
computing a fourth projection matrix for the second frame based on the second projection matrix for the first frame, the projection mask, a result of the second calculation from the second layer, and the third layer being included in the set of key layers; and
generating label data for the video based on the fourth projection matrix, wherein the label data includes object tracking data for the first frame and the second frame.
2. The method of claim 1 , wherein:
the projection mask indicates regions where parameters of the third projection matrix have more than or equal to a threshold difference from corresponding parameters of the first projection matrix.
3. The method of claim 1 , further comprising:
selecting parameters used as input to the third layer from the second projection matrix and the third projection matrix based on the projection mask, wherein the projection mask indicates regions where parameters of the third projection matrix have less than or equal to a threshold difference from corresponding parameters of the first projection matrix.
4. The method of claim 1 , further comprising:
determining that a fourth layer of the neural network is a key layer; and
using output features of the third layer from the first frame as input for the second frame to the fourth layer of the neural network based on the determination that the third layer and fourth layer are key layers, wherein
one or more layers of the neural network are present between the third layer and the fourth layer which are not key layers.
5. The method of claim 1 , further comprising:
performing a first non-linear activation operation on the first projection matrix to obtain first output features of the first layer of the neural network for the first frame.
6. The method of claim 1 , further comprising:
performing a first non-linear activation operation on the second third projection matrix to obtain second output features of the first layer of the neural network for the second frame.
7. The method of claim 1 , wherein:
the first convolution operation comprises a linear operation.
8. The method of claim 1 , further comprising:
compressing the first projection matrix to generate a first compressed projection matrix which has fewer parameters than the first projection matrix;
storing the first compressed projection matrix in memory during processing of the first frame; and
retrieving the first compressed projection matrix from the memory during processing of the second frame to perform the comparing.
9. The method of claim 1 , further comprising:
storing the first projection matrix in memory during processing of the first frame; and
retrieving the first projection matrix from the memory during processing of the second frame to compute the fourth projection matrix.
10. A method comprising:
receiving a video comprising a plurality of video frames;
determining that a first layer and a second layer of a neural network are in a set of key layers;
determining that a set of non-key layers comprises one or more layers of the neural network between the first layer and the second layer;
computing a first projection matrix by performing a first convolution operation on first input features of a first frame from the plurality of video frames at a first layer of neural network;
computing a second projection matrix by performing the first convolution operation on second input features of a second frame of the plurality of video frames at the first layer;
comparing the first projection matrix to the second projection matrix to obtain a projection mask;
computing a third projection matrix by performing, on a first portion of third input features of the second frame based on the projection mask and based on the first layer and the second layer being key layers, mathematical operations comprising convolution at the set of non-key layers;
determining that a third layer of the neural network is a key layer;
using output features of the second layer from the first frame as input for the third layer of the neural network based on the determination that the second layer and third layer are key layers; and
generating label data for the video based on output of the third layer, wherein
the label data includes the location(s) of one or more regions relevant to computer vision or image processing tasks within the plurality of video frames.
11. The method of claim 10 , further comprising:
compressing the first projection matrix to obtain a first compressed projection matrix; and
compressing the second projection matrix to obtain a second compressed projection matrix, wherein the comparison is between the first compressed projection matrix and the second compressed projection matrix.
12. The method of claim 11 , further comprising:
performing a second convolution operation on parameters of the first projection matrix corresponding to the second portion based on the projection mask.
13. The method of claim 10 , further comprising:
determining a second portion of the third input features based on the projection mask, wherein the projection mask indicates that parameters of the second projection matrix corresponding to the second portion have less than or equal to a threshold difference from corresponding parameters of the first projection matrix; and
using the second portion of the third input features as an input to the second layer.
14. The method of claim 10 , further comprising:
storing the first projection matrix in memory during processing of the first frame; and
retrieving the first projection matrix from the memory during processing of the second frame to compute the third projection matrix.
15. An apparatus comprising an electronic device comprising a microprocessor and camera, wherein
the microprocessor is configured to process images using an algorithm comprising:
a first key layer configured to compute a first projection matrix by performing a first convolution operation on first input features of a first frame from a plurality of video frames, and to compute a second projection matrix by performing the first convolution operation on second input features of a second frame of the plurality of video frames at the first layer;
a mask component configured to generate a projection mask by comparing the first projection matrix to the second projection matrix;
a set of one or more non-key layers after the first key layer configured to compute a third projection matrix by mathematical operations comprising performing a second convolution operation on a first portion of third input features of the second frame at based on the projection mask and based on the first key layer being a key layer;
a second key layer after the set of non-key layers which is configured to perform a third convolution operation on the third input features of the second frame which are not in the first portion and on the output of the set of non-key layers based on the first key layer being a key layer; and
generating label data for the video based on the third projection matrix, and wherein the label data includes the location(s) of one or more regions relevant to computer vision or image processing tasks within the plurality of video frames.
16. The apparatus of claim 15 , wherein the algorithm further comprises:
a compression component configured to compress the first projection matrix to obtain a first compressed projection matrix, and the to compress the second projection matrix to obtain a second compressed projection matrix.
17. The apparatus of claim 15 , further comprising:
a memory configured to store the first projection matrix.