Sparsifying layers of neural network models to different levels
A method of pruning a pre-trained model comprises the steps of (a) constructing a stochastic super net and (b) training the stochastic super net to determine a particular candidate block selection that provides an optimal level of sparsity for each of the layers based upon a cost function. The stochastic super net generally represents a layer-wise search space with a fixed macro-architecture. A number of layers of the macro-architecture and input/output dimensions of each of the layers of the macro-architecture are essentially the same as the pre-trained model. Each layer comprises a plurality of candidate blocks. A sparsity of each of the candidate blocks in a respective layer is different. A training dataset used to train the pre-trained model is used to train the stochastic super net.
1 . A method of pruning a pre-trained neural network model utilizing kernel sparsification comprising the steps of: constructing a stochastic super net representing a layer-wise search space with a fixed macro-architecture, wherein
(i) the fixed macro-architecture has a number of layers that match the number of layers of the pre-trained neural network model and each of the number of layers of the fixed macro-architecture has input/output dimensions that match the input/output dimensions of a corresponding layer of the pre-trained neural network model,
(ii) each layer of the fixed macro-architecture comprises a plurality of candidate blocks,
(iii) each of the plurality of candidate blocks of each layer of the fixed macro-architecture has (a) an architecture that matches the architecture of the corresponding layer of the pre-trained neural network model, (b) a respective kernel sparsity level that is different from the respective kernel sparsity levels of the other candidate blocks in the same layer of the fixed macro-architecture and (c) an operator that is the same as an activation function of the corresponding layer of the pre-trained neural network model, wherein the respective kernel sparsity level of each of the candidate blocks in each of the layers of the fixed macro-architecture represents a portion of coefficients removed in a respective kernel of each of the plurality of candidate blocks in each of the layers of the fixed macro-architecture relative to the corresponding layer of the pre-trained neural network model;
training the stochastic super net, using a training dataset used to train the pre-trained neural network model; and
generating a pruned version of the pre-trained neural network model by determining a particular candidate block selection in each of the layers of the fixed macro-architecture that provides an optimal level of kernel sparsity for each layer of the pruned version of the pre-trained neural network model based upon a cost function, wherein the pruned version of the pre-trained neural network is compiled using a plurality of dedicated hardware engines to enable an execution of a directed acyclic graph on predetermined hardware of an edge device.
2 . The method according to claim 1 , wherein different layers of said pruned version of the pre-trained neural network model are sparsified to different levels of kernel sparsity.
3 . The method according to claim 1 , wherein said method generates a sparsified neural network via a differentiable neural network search.
4 . The method according to claim 1 , further comprising the step of
creating a latency lookup table offline to estimate an overall latency of a network based on a number of multiply accumulate operations of each operator, wherein (i) said cost function includes a first term denoting a cross-entropy loss of the respective kernel sparsity level and a second term denoting a latency of the respective kernel sparsity level and (ii) said latency of said second term of said cost function is determined in response to a summation of said overall latency of each of said layers at said respective kernel sparsity level.
5 . The method according to claim 4 , wherein said second term denoting the latency of the respective kernel sparsity level accounts for performance on said predetermined hardware of said edge device.
6 . The method according to wherein the respective kernel sparsity levels of the plurality of candidate blocks of each respective layer of the fixed macro-architecture are configured to cover a respective range of kernel sparsity levels for the respective layer in response to both (a) hardware resources and (b) a model and configuration of a chip of said predetermined hardware of said edge device.
7 . The method according to wherein said step of training the stochastic super net comprises a plurality of epochs, and in each epoch a kernel or weight of an operator of each of the plurality of candidate blocks is trained with 80% of said training dataset and an architecture distribution parameter of each of the plurality of candidate blocks is trained on a remaining 20% of said training dataset.
8 . The method according to claim 1 , wherein only layers with convolutional operators are pruned utilizing kernel sparsification.
9 . The method according to claim 1 , wherein fully connected layers and layers with convolutional operators are pruned utilizing kernel sparsification.
10 . An apparatus comprising:
a camera configured to capture a sequence of images; and
a processing circuit configured to perform a computer vision operation on said sequen of images utilizing a sparsified neural network, wherein
(A) said sparsified neural network is generated by (a) constructing a stochastic super net representing a layer-wise search space with a fixed macro-architecture, where (i) the fixed macro-architecture has a number of layers that match a number of layers of a full density pre-trained neural network model and each of the number of layers of the fixed macro-architecture has input/output dimensions that match the input/output dimensions of a corresponding layer of the full density pre-trained neural network model, (ii) each layer of the fixed macro-architecture comprises a plurality of candidate blocks, (iii) each of the plurality of candidate blocks of each layer of the fixed macro-architecture has (a) an architecture that matches the architecture of the corresponding layer of the full density pre-trained neural network model, (b) a respective kernel sparsity level that is different from the respective kernel sparsity levels of the other candidate blocks in the same layer of the fixed macro-architecture and (c) an operator that is the same as an activation function of the corresponding layer of the pre-trained neural network model, wherein the respective kernel sparsity level of each of the candidate blocks in each of the layers of the fixed macro-architecture represents a portion of coefficients removed in a respective kernel of each of the plurality of candidate blocks in each of the layers of the fixed macro-architecture relative to the corresponding layer of the full density pre-trained neural network model, (b) training the stochastic super net, using a training dataset used to train the full density pre-trained neural network model, and (c) determining a layer-wise candidate block assignment that provides an optimal level of kernel sparsity for each layer of the sparsified neural network based upon a cost function,
(B) said processing circuit comprises a plurality of dedicated hardware engines, and
(C) said layer-wise candidate block assignment of said sparsified neural network is compiled using said plurality of dedicated hardware engines to enable execution of a directed acyclic graph on predetermined hardware of an edge device.
11 . The method according to claim 1 , wherein (i) a creation of parallel blocks is restricted to convolution layers and fully connected layers of said of said fixed macro-architecture and (ii) densities of said parallel blocks are determined according to an acceptable density range based on both (a) hardware resources and (b) a model and configuration of a chip of said predetermined hardware of said edge device.
12 . The apparatus according to claim 11 , wherein said apparatus implements said edge device.
13 . The apparatus according to claim 10 , wherein said camera is implemented as at least one of a traffic camera, an automatic number plate recognition camera, a dome camera, a bullet camera, an automatic teller machine camera, an access control camera, a vehicle camera, a doorbell camera, a battery-powered camera, an indoor camera, and an outdoor camera.
14 . The apparatus according to claim 10 , wherein said sparsified neural network is configured for one or more of feature detection, feature recognition, object detection, multi-object detection, object recognition, and face recognition.
15 . A non-transitory computer readable storage medium embodying computer executable instructions, which when executed cause a computer to perform a method of pruning a pre-trained neural network model utilizing kernel sparsification comprising the steps of:
constructing a stochastic super net representing a layer-wise search space with a fixed macro-architecture, wherein (i) the fixed macro-architecture has a number of layers that match the number of layers of the pre-trained neural network model and each of the number of layers of the fixed macro-architecture has input/output dimensions that match the input/output dimensions of a corresponding layer of the pre-trained neural network model, (ii) each layer of the fixed macro-architecture comprises a plurality of candidate blocks, (iii) each of the plurality of candidate blocks of each layer of the fixed macro-architecture has (a) an architecture that matches the architecture of the corresponding layer of the pre-trained neural network model, (b) a respective kernel sparsity level that is different from the respective kernel sparsity levels of the other candidate blocks in the same layer of the fixed macro-architecture and (c) an operator that is the same as an activation function of the corresponding layer of the pre-trained neural network model, wherein the respective kernel sparsity level of each of the candidate blocks in each of the layers of the fixed macro-architecture represents a portion of coefficients removed in a respective kernel of each of the plurality of candidate blocks in each of the layers of the fixed macro-architecture relative to the corresponding layer of the pre-trained neural network model;
training the stochastic super net, using a training dataset used to train the pre-trained neural network model; and
generating a pruned version of the pre-trained neural network model by determining a layer-wise candidate block assignment that provides an optimal level of kernel sparsity for each layer of the pruned version of the pre-trained neural network model based upon a cost function, wherein the pruned version of the pre-trained neural network is compiled using a plurality of dedicated hardware engines to enable execution of a directed acyclic graph on predetermined hardware of an edge device.
16 . The non-transitory computer readable storage medium according to claim 15 , wherein said computer executable instructions provide flexibility to sparsify different layers to different levels of kernel sparsity.
17 . The non-transitory computer readable storage medium according to claim 15 , wherein said computer executable instructions generate a sparsified neural network via a differentiable neural network search.
18 . The non-transitory computer readable storage medium according to claim 15 , wherein said cost function includes a latency term that accounts for performance on said predetermined hardware of said edge device.
19 . The non-transitory computer readable storage medium according to claim 15 , wherein said step of training the stochastic super net comprises a plurality of epochs, and in each epoch a kernel or weight of an operator of each of the plurality of candidate blocks is trained with a first portion of said training dataset and an architecture distribution parameter of each of the plurality of candidate blocks is trained on a second portion of said training dataset.
20 . The non-transitory computer readable storage medium according to claim 15 , wherein said step of generating a pruned version of the pre-trained neural network model comprises one or more of pruning each of said layers utilizing kernel sparsification, pruning only layers with convolutional operators utilizing kernel sparsification, or pruning only fully connected layers and layers with convolutional operators utilizing kernel sparsification.