Method and electronic device for quantizing DNN model
Various embodiments of the disclosure disclose a method for quantizing a Deep Neural Network (DNN) model in an electronic device. The method includes: estimating, by the electronic device, an activation range of each layer of the DNN model using self-generated data (e.g. retro image, audio, video, etc.) and/or a sensitive index of each layer of the DNN model; quantizing, by the electronic device, the DNN model based on the activation range and/or the sensitive index; and allocating, by the electronic device, a dynamic bit precision for each channel of each layer of the DNN model to quantize the DNN model.
1 . A method for quantizing a Deep Neural Network (DNN) model in an electronic device comprising a quantization engine, wherein the quantization engine comprises processing circuitry and/or executable program instructions, operably connected to the memory and the processor, the method comprising:
estimating, by the quantization engine, at least one of an activation range of each layer of the DNN model using self-generated data and a sensitive index of each layer of the DNN model; and
quantizing, by the quantization engine, the DNN model based on the at least one of the activation range and the sensitive index
wherein estimating, by the quantization engine, the activation range of each layer of the DNN model using the self-generated data comprises:
determining, by the quantization engine, a plurality of random images, wherein each random image of the plurality of random images comprises uniform distribution data across the images;
passing, by the quantization engine, each random image into the DNN model;
determining, by the quantization engine, weight distributions of the DNN model for each random image after each layer of the DNN model;
determining, by the quantization engine, layer statistics of the DNN model for each random image after each layer of the DNN model, wherein the layer statistics of the DNN model comprises at least one of a mean and a variance;
determining, by the quantization engine, a difference between pre-stored layer statistics of the DNN model and the determined layer statistics of the DNN model;
determining, by the quantization engine, whether the difference is less than a threshold; and
performing, by the quantization engine, one of:
generating the data using at least one of the layer statistics of the DNN model and the weight distributions of the DNN model in response to determining that the difference is less than the threshold, or
executing back propagation in the DNN model in response to determining that the difference is greater than or equal to the threshold.
2 . The method as claimed in claim 1 , wherein the self-generated data is generated based on at least one of layer statistics of the DNN model and weight distributions of the DNN model.
3 . The method as claimed in claim 1 , wherein the DNN model quantizes weights and activation at lower bit precision without access to at least one of training dataset and validation dataset to obtain a compression of the DNN model and a fast inference of the DNN model.
4 . The method as claimed in claim 1 , wherein the self-generated data is a plurality of retro data images, wherein the plurality of retro data images are equivalent or represent all features of the DNN model.
5 . The method as claimed in claim 1 , wherein a Z-score provides a difference between two distributions.
6 . The method as claimed in claim 1 , wherein the self-generated data is a plurality of retro data images, wherein the plurality of retro data images are equivalent or represent all features of the DNN model.
7 . The method as claimed in claim 1 , wherein estimating, by the quantization engine, the sensitive index of each layer of the DNN model comprises:
determining, by the quantization engine, an accuracy using per-channel quantization and per-tensor quantization schemes at each layer of the DNN model; and
determining, by the quantization engine, a sensitive index of each layer corresponding to per-channel quantization and per-tensor quantization.
8 . The method as claimed in claim 1 , wherein quantizing, by the quantization engine, the DNN model based the sensitive index comprises:
determining, by the quantization engine, an optimal quantization for each layer of the DNN model based on the sensitive index to quantize the DNN model, wherein the sensitive index comprises a minimum sensitivity; and
applying, by the quantization engine, the optimal quantization for each layer of the DNN model.
9 . The method as claimed in claim 1 , wherein the sensitive index is determined a using a Kullback-Leibler Divergence.
10 . The method as claimed in claim 8 , wherein the optimal quantization comprises at least one of per-channel quantization, and per-tensor quantization for the DNN model.
11 . The method as claimed in claim 1 , wherein the sensitive index is used to reduce a search space of each layer of the DNN model from exponential to linear, wherein n is a number of layers in the DNN model.
12 . The method as claimed in claim 1 , wherein the method comprises:
allocating, by the quantization engine, a dynamic bit precision for each channel of each layer of the DNN model, wherein the allocated dynamic bit precision minimizes and/or reduces overall quantization noise of each layer.
13 . An electronic device configured to quantize a Deep Neural Network (DNN) model, the electronic device comprising:
a memory;
a processor comprising processing circuitry; and
a quantization engine comprising processing circuitry and/or executable program instructions, operably connected to the memory and the processor, configured to:
estimate at least one of an activation range of each layer of the DNN model using self-generated data and a sensitive index of each layer of the DNN model;
determine a plurality of random images, wherein each random image of the plurality of random images comprises uniform distribution data across the images;
pass each random image into the DNN model;
determine weight distributions of the DNN model for each random image after each layer of the DNN model;
determine layer statistics of the DNN model for each random image after each layer of the DNN model, wherein the layer statistics of the DNN model comprises at least one of a mean and a variance;
determine a difference between pre-stored layer statistics of the DNN model and the determined layer statistics of the DNN model;
determine whether the difference is less than a threshold;
perform, one of:
generate the data using at least one of the layer statistics of the DNN model and the weight distributions of the DNN model in response to determining that the difference is less than the threshold; or
execute back propagation in the DNN model in response to determining that the difference is greater than or equal to the threshold; and
quantize the DNN model based on the at least one of the activation range and the sensitive index.