Low precision and coarse-to-fine dynamic fixed-point quantization design in convolution neural network
After inputting input data to a floating pre-trained convolution neural network to generate floating feature maps for each layer of the floating pre-trained CNN model, a statistical analysis on the floating feature maps is performed to generate a dynamic quantization range for each layer of the floating pre-trained CNN model. Based on the obtained quantization range for each layer, the proposed quantization methodologies quantize the floating pre-trained CNN model to generate the scalar factor of each layer and the fractional bit-width of a quantized CNN model. It enables the inference engine performs low-precision fixed-point arithmetic operations to generate a fixed-point inferred CNN model.
1. A method of quantizing a floating pre-trained convolution neural network (CNN) model comprising:
inputting input data to the floating pre-trained CNN model to generate floating feature maps for each layer of the floating pre-trained CNN model;
inputting the floating feature maps to a statistical analysis simulator to generate a dynamic quantization range for each layer of the floating pre-trained CNN model; and
quantizing the floating pre-trained CNN model according to the dynamic quantization range for each layer of the floating pre-trained CNN model to generate a quantized CNN model, a scalar factor of each layer of the floating pre-trained CNN model, and a fractional bit-width of the quantized CNN model, wherein quantizing the floating pre-trained CNN model comprises:
acquiring a plurality of weights of each layer of the floating pre-trained CNN model;
setting the scalar factor of each layer of the floating pre-trained CNN model according to a maximum weight of the plurality of weights and a minimum weight of the plurality of weights;
applying the scalar factor of each layer of the floating pre-trained CNN model to an activation vector at each layer of the floating pre-trained CNN model; and
minimizing a quantization error of each layer of the quantized CNN model according to the scalar factor by using a minimum mean square error approach as
Min
{
(
∑
i
=
1
M
(
Q
(
s
(
l
)
·
x
i
(
l
)
)
-
(
s
(
l
)
·
x
i
(
l
)
)
)
2
}
,
wherein s (l) is the scalar factor at an l-th layer, x i (l) represents output features in an i-th channel at the l-th layer, Q( ) is a quantization function, and M is a total number of channels;
wherein the scalar factor of each layer of the floating pre-trained CNN model is associated with a quantization bit-width and the dynamic quantization range when quantizing the floating pre-trained CNN model.
2. The method of claim 1 , wherein the quantizing the floating pre-trained CNN model further comprises:
performing a first quantization over the floating pre-trained CNN model according to the dynamic quantization range for each layer of the floating pre-trained CNN model to generate a first CNN model; and
performing a second quantization on all channels of convolution (CONV) layers and/or fully-connected (FC) layers of the first CNN model to generate the quantized CNN model.
3. The method of claim 2 , wherein performing the first quantization over the floating pre-trained CNN model according to the dynamic quantization range for each layer of the floating pre-trained CNN model to generate the first CNN model comprises:
performing an activation quantization process, a bias quantization process, and a weight quantization process to generate the first CNN model.
4. The method of claim 3 , wherein performing the activation quantization process, the bias quantization process and the weight quantization process to generate the first CNN model comprises:
performing the activation quantization process over the floating pre-trained CNN model to generate quantized activations;
performing the bias quantization process over bias data of the floating pre-trained CNN model to generate a quantized bias; and
performing the weight quantization process over the weight data of the floating pre-trained CNN model to generate quantized weights.
5. The method of claim 2 , wherein performing the second quantization on all channels of convolution (CONV) layers and/or fully-connected (FC) layers of the first CNN model to generate the quantized CNN model comprises:
performing a weight per channel quantization process over all channels of convolution (CONV) layers and/or fully-connected (FC) layers of the first CNN model to generate the quantized CNN model which comprises a quantized weight per channel.
6. The method of claim 1 , wherein the input data is multimedia data.