Data processing method and device for bit-number reduction in a neural network by establishing bit upper bound and bit lower bound using distribution information
A data processing method used in neural network computing is provided. During a training phase of a neural network model, a feedforward procedure based on a calibration data is performed to obtain distribution information of a feedforward result for at least one layer of the neural network model. During the training phase of the neural network model, a bit upper bound of a partial sum is generated based on the distribution information of the feedforward result. During an inference phase of the neural network model, a bit-number reducing process is performed on an original operation result of an input data and a weight for the neural network model according to the bit upper bound of the partial sum to obtain an adjusted operation result.
1. A data processing method used in neural network computing, comprising:
during a training phase of a neural network model, performing a feedforward procedure based on a calibration data to obtain a distribution information of a feedforward result for at least one layer of the neural network model;
during the training phase of the neural network model, generating a bit upper bound of a partial sum based on the distribution information of the feedforward result; and
generating a bit lower bound of the partial sum by deducting a bit width of an accumulator from the bit upper bound of the partial sum;
wherein during an inference phase of the neural network model, a bit-number reducing process is performed on an original operation result of an input data and a weight for the neural network model according to the bit upper bound of the partial sum to obtain an adjusted operation result.
2. The method according to claim 1 , wherein the neural network model used for performing feedforward procedure based on the calibration data to obtain the distribution information of the feedforward result is a pre-trained neural network model.
3. The method according to claim 1 , wherein the distribution information of the feedforward result at least comprises a mean and a standard deviation, the bit upper bound of the partial sum is determined according to the mean and the standard deviation.
4. The method according to claim 3 , wherein the bit upper bound of the partial sum is related to a binary logarithm of a value which is the mean plus N times of the standard deviation, N is an integer.
5. The method according to claim 1 , further comprising:
training the neural network model again by using the bit upper bound of the partial sum to perform saturation procedure to a training operation result of a training data through a training unit.
6. The method according to claim 1 , further comprising:
training the neural network model again by using the bit upper bound and the bit lower bound of the partial sum to perform saturation procedure to a training operation result of a training data.
7. The method according to claim 1 , further comprising:
measuring accuracy by using an evaluation unit with testing data, and training the neural network model again by using the bit upper bound and the bit lower bound of the partial sum to perform saturation procedure to a training operation result of a training data when the accuracy is lower than a threshold.
8. The method according to claim 1 , wherein the input data and the weight are in a form of integer, the original operation result, the adjusted operation result are in the form of integer, and the original operation result of the input data and the weight is the result of at least one of convolution operation, multiplication operation, matrix multiplication operation, dot product operation, and accumulation operation for the input data and the weight.
9. The method according to claim 1 , wherein a value of the bit upper bound is equal or smaller than a bit width of an accumulator in hardware.
10. A data processing device used in neural network computing, comprising:
a first operation circuit, for receiving an input data and a weight for a neural network model and outputting a first operation result;
an adjusting circuit, for performing bit-number reduced operation on the first operation result according to a bit upper bound of a partial sum to obtain a second operation result during an inference phase of the neural network model;
a second operation circuit, for receiving the second operation result and a third operation result to generate a fourth operation result; and
a storing circuit, for storing the fourth operation result;
wherein the bit upper bound of the partial sum is generated based on distribution information of a feedforward result obtained by performing feedforward procedure based on calibration data for at least one layer of the neural network model during a training phase of the neural network model;
wherein the adjusting circuit comprises a first saturation circuit for performing saturation procedure on the first operation result according to the bit upper bound of the partial sum;
wherein the adjusting circuit further comprises a shifting circuit for performing shifting operation on an output of the first saturation circuit, the shifting operation is performed according to a bit lower bound of the partial sum which is generated by deducting a bit width of an accumulator from the bit upper bound of the partial sum.
11. The data processing device according to claim 10 , wherein the first operation circuit comprises a multiplier, the first operation result is the multiplication of the input data and the weight.
12. The data processing device according to claim 10 , wherein the distribution information of the feedforward result at least comprises a mean and a standard deviation, and the bit upper bound of the partial sum is determined according to the mean and the standard deviation.
13. The data processing device according to claim 12 , wherein the bit upper bound of the partial sum is related to a binary logarithm of a value which is the mean plus N times of the standard deviation, Nis an integer.
14. The data processing device according to claim 10 , wherein the input data and the weight are in a form of integer, the first operation result, the second operation result, the third operation result, and the fourth operation result are in the form of integer.
15. The data processing device according to claim 10 , wherein a value of the bit upper bound is equal or smaller than a bit width of an accumulator in hardware.
16. The data processing device according to claim 10 , wherein the second operation circuit comprises an adder which adding the second operation result and the third operation result.
17. The data processing device according to claim 16 , wherein the second operation circuit further comprises a second saturation circuit which performs saturation procedure on an output of the adder.