Neural network construction method and apparatus having average quantization mechanism
The present invention discloses a neural network construction method having average quantization mechanism that includes steps outlined below. A weight combination included in each of network layers of a neural network is retrieved. A loss function is generated according to the weight combination of all the network layers and target values. Corresponding to each network layers, a Gini coefficient of the weight combination is calculated and the Gini coefficients corresponding to all the network layers are accumulated as a regularized correction term. The loss function and the regularized correction term are merged as a regularized loss function to perform training on the neural network accordingly to generate a trained weight combination of each of the network layers. A quantization is performed on the trained weight combination of each of the network layers to generate a quantized neural network, in which each of the network layers thereof includes the trained weight combination.
1. A neural network construction method having average quantization mechanism, comprising:
retrieving a weight combination comprised in each of network layers of a neural network, wherein the weight combination comprises a plurality of floating-point weights;
generating a loss function according to the weight combination of all the network layers and a plurality of target values, wherein the loss function is a first function of the weight combination and the target values;
corresponding to each of the network layers, calculating a Gini coefficient of the weight combination and accumulating the Gini coefficient of each of the network layers as a regularized correction term, wherein the regularized correction term is a second function of the weight combination;
merging the loss function and the regularized correction term to generate a regularized loss function to perform training on the neural network according to the regularized loss function to keep modifying the floating-point weights in the weight combination, so as to generate a trained weight combination of each of the network layers;
performing quantization on the trained weight combination of each of the network layers to generate a quantized neural network, in which each of the network layers of the quantized neural network comprises the quantization weight combination, wherein the quantization weight combination comprises a plurality of integer weights, and a first distribution of the floating-point weights in the trained weight combination is even such that a second distribution of the integer weights in the quantization weight combination is even; and
implementing the quantized neural network by an embedded system chip.
2. The neural network construction method of claim 1 , further comprising:
multiplying the regularized correction term by an order correction parameter such that the regularized correction term and the loss function have the same order.
3. The neural network construction method of claim 1 , wherein the Gini coefficient has a larger value when a distribution of the weight combination is more uneven and has a smaller value when the distribution of the weight combination is more even.
4. A neural network construction apparatus having average quantization mechanism comprising:
a storage circuit configured to store a computer executable command; and
a processing circuit configured to retrieve and execute the computer executable command to execute a neural network construction method, comprising:
retrieving a weight combination comprised in each of network layers of a neural network, wherein the weight combination comprises a plurality of floating-point weights;
generating a loss function according to the weight combination of all the network layers and a plurality of target values, wherein the loss function is a first function of the weight combination and the target values;
corresponding to each of the network layers, calculating a Gini coefficient of the weight combination and accumulating the Gini coefficient of each of the network layers as a regularized correction term, wherein the regularized correction term is a second function of the weight combination;
merging the loss function and the regularized correction term to generate a regularized loss function to perform training on the neural network according to the regularized loss function to keep modifying the floating-point weights in the weight combination, so as to generate a trained weight combination of each of the network layers;
performing quantization on the trained weight combination of each of the network layers to generate a quantized neural network, in which each of the network layers of the quantized neural network comprises the quantization weight combination, wherein the quantization weight combination comprises a plurality of integer weights, and a first distribution of the floating-point weights in the trained weight combination is even such that a second distribution of the integer weights in the quantization weight combination is even; and
implementing the quantized neural network by an embedded system chip.
5. The neural network construction apparatus of claim 4 , wherein the neural network construction method further comprising:
multiplying the regularized correction term by an order correction parameter such that the regularized correction term and the loss function have the same order.
6. The neural network construction apparatus of claim 4 , wherein the Gini coefficient has a larger value when a distribution of the weight combination is more uneven and has a smaller value when the distribution of the weight combination is more even.