Granted Patent B2

US 12,518,148 · App. 17/254,372 · Granted Jan 6, 2026

Inventors: Shaoli Liu (Anhui, CN); Di Huang (Anhui, CN); Xishan Zhang (Anhui, CN); Yao Zhang (Anhui, CN)

Assignee: ANHUI CAMBRICON INFORMATION TECHNOLOGY CO., LTD.

G06N3/063G06N3/08

Abstract

Claims (80)

1 . A neural network quantization method, comprising:

determining a plurality of pieces of data to be quantized in target data of a layer in a neural network to be quantized, wherein each piece of data to be quantized is a subset of the target data, the target data is data to be operated and quantized in the layer to be quantized, and the data to be operated and quantized includes at least one of an input neuron, a weight, a bias, and a gradient;

and wherein a quantity of the plurality of pieces of data depends on one or more dimensions of the target data and the quantity of the plurality of pieces of data is inversely correlated to a processing capability of a device implementing the neural network quantization;

determining, for each piece of data, a corresponding quantization parameter based on a pre-determined correspondence, wherein the corresponding quantization parameter comprises one or more of the following: a decimal point position, a scaling factor, or an offset;

quantizing each piece of data to be quantized according to the corresponding quantization parameter to obtain a piece of quantized data corresponding to each piece of data to be quantized, wherein the plurality of pieces of data are quantized in parallel;

obtaining a quantization result of the target data according to the piece of quantized data corresponding to each piece of data to be quantized, so that an operation is performed in the layer to be quantized according to the quantization result of the target data;

determining a quantization error corresponding to the target data, wherein the determining of the quantization error comprises:

determining a difference between the plurality of pieces of data to be quantized and a plurality of pieces of de-quantized data of the plurality of quantized data; or

determining a difference as a total quantization interval over the plurality of pieces of data to be quantized;

wherein the quantization error is an average of the differences over a sum of the plurality of pieces of data to be quantized; and

according to the quantization error and an error threshold determined based on an empirical value, adjusting a data bit width of each piece of data to be quantized to obtain an adjusted bit width corresponding to each piece of data to be quantized.

2 . The neural network quantization method of claim 1 , wherein the layer to be quantized is a convolution layer, the target data is an input neuron, and the determining a plurality of pieces of data to be quantized in the target data of the layer to be quantized includes:

in the input neuron of the convolution layer, determining a plurality of pieces of data to be quantized corresponding to a convolution kernel according to a dimension and stride of the convolution kernel, where the dimension of the convolution kernel includes height, width, and a count of channels.

3 . The neural network quantization method of claim 1 , wherein the determining the plurality of pieces of data to be quantized in the target data of the layer to be quantized includes:

determining a plurality of pieces of data to be quantized in the target data of the layer to be quantized according to a dimension of the target data, where the dimension of the target data includes a count of batches, channel, height, and width, wherein the determining a plurality of pieces of data to be quantized in the target data of the layer to be quantized according to the dimension of the target data includes:

determining one or more batches of data in the target data of the layer to be quantized as one piece of data to be quantized; or

determining one or more channels of data in the target data of the layer to be quantized as one piece of data to be quantized.

4 . The neural network quantization method of claim 1 , wherein the determining a plurality of pieces of data to be quantized in the target data of the layer to be quantized includes:

according to a real-time processing capability of a device running the neural network, determining a plurality of pieces of data to be quantized in the target data of the layer to be quantized, where a size of each piece of data to be quantized is positively correlated with the real-time processing capability.

5 . The neural network quantization method of claim 1 , further comprising:

computing a corresponding quantization parameter according to each piece of data to be quantized and a corresponding data bit width, wherein the computing the corresponding quantization parameter according to each piece of data to be quantized and the corresponding data bit width includes:

when the quantization parameter does not include an offset of each piece of data to be quantized, obtaining a first type of point position of each piece of data to be quantized according to a maximum of an absolute value of each piece of data to be quantized and the corresponding data bit width; or

when the quantization parameter does not include an offset of each piece of data to be quantized, obtaining a maximum of the piece of quantized data according to each piece of data to be quantized and the corresponding data bit width, obtaining a first type of scaling factor of each piece of data to be quantized according to the maximum of the absolute value of each piece of data to be quantized and the maximum of the piece of quantized data; or

when the quantization parameter includes an offset of each piece of data to be quantized, obtaining a second type of point position of each piece of data to be quantized according to a maximum and a minimum of each piece of data to be quantized and the corresponding data bit width; or

when the quantization parameter includes an offset of each piece of data to be quantized, obtaining the maximum of the piece of quantized data according to each piece of data to be quantized and the corresponding data bit width, and obtaining a second type of scaling factor of each piece of data to be quantized according to the maximum and the minimum of each piece of data to be quantized, and the maximum of the piece of quantized data.

6 . The neural network quantization method of claim 1 , further comprising:

updating the data bit width corresponding to each piece of data to be quantized to a corresponding adjusted bit width, and computing a corresponding adjusted quantization parameter according to each piece of data to be quantized and the corresponding adjusted bit width to quantize each piece of data to be quantized according to the corresponding adjusted quantization parameter.

7 . The neural network quantization method of claim 6 , wherein the adjusting the data bit width of each piece of data to be quantized to obtain an adjusted bit width corresponding to each piece of data to be quantized according to the quantization error and an error threshold corresponding to each piece of data to be quantized includes when the quantization error is greater than a first error threshold, increasing the corresponding data bit width to obtain the corresponding adjusted bit width; and

further comprising:

computing an adjusted quantization error of each piece of data to be quantized according to the piece of data to be quantized and the corresponding adjusted bit width;

continuing to increase the corresponding adjusted bit width according to the adjusted quantization error and the first error threshold until the adjusted quantization error is less than or equal to the first error threshold;

computing the adjusted quantization error of the data to be quantized according to the adjusted bit width and the piece of data to be quantized; and

continuing to decrease the corresponding adjusted bit width according to the adjusted quantization error and the second error threshold until the adjusted quantization error obtained according to the adjusted bit width and the piece of data to be quantized is greater than or equal to the second error threshold

wherein the adjusting the data bit width of each piece of data to be quantized to obtain an adjusted bit width corresponding to each piece of data to be quantized according to the quantization error and an error threshold corresponding to each piece of data to be quantized includes when the quantization error is less than a second error threshold, increasing the corresponding data bit width to obtain the corresponding adjusted bit width, where the second error threshold is less than the first error threshold.

8 . The neural network quantization method of claim 1 , wherein, during a fine-tuning stage and/or training stage of a neural network operation, the method further comprises:

obtaining a variation range of the data to be quantized in a current iteration and historical iterations, where the historical iterations are iterations before the current iteration; and

according to the variation range of the data to be quantized, determining a target iteration interval corresponding to the data to be quantized to enable the layer to be quantized to update the quantization parameter of the data to be quantized according to the target iteration interval, where the target iteration interval includes at least one iteration.

9 . The neural network quantization method of claim 8 , further comprising:

according to the data bit width of the data to be quantized in the current iteration, determining the data bit width of the data to be quantized in the iterations within the target iteration interval to enable the neural network to determine the quantization parameter; and

according to a point position of the data to be quantized in the current iteration, determining the point position of the data to be quantized in the iterations within the target iteration interval, where the point position includes the first type of point position and/or the second type of point position.

10 . A neural network quantization device comprising one or more processing circuits, wherein for any layer to be quantized in a neural network, the device comprises:

a data determination module implemented on the one or more processing circuits and configured to determine a plurality of pieces of data to be quantized in target data of a layer to be quantized, wherein each piece of data to be quantized is a subset of the target data, the target data is any kind of data to be operated and quantized in the layer to be quantized, and the data to be operated includes at least one of an input neuron, a weight, a bias, and a gradient, and wherein a quantity of the plurality of pieces of data depends on one or more dimensions of the target data and the quantity of the plurality of pieces of data is inversely correlated to a processing capability of a device implementing the neural network quantization;

a data quantization module implemented on the one or more processing circuits and configured to determine, for each piece of data, a corresponding quantizing parameter based on a pre-determined correspondence, wherein the corresponding quantization parameter comprises one or more of the following: a decimal point position, a scaling factor, or an offset, and quantize each piece of data to be quantized according to the corresponding quantization parameter to obtain a piece of quantized data corresponding to each piece of data to be quantized, wherein the plurality of pieces of data are quantized in parallel;

a data operation module implemented on the one or more processing circuits and configured to obtain a quantization result of the target data according to the piece of quantized data corresponding to each piece of data to be quantized, so that an operation may be performed in the layer to be quantized according to the quantization result of the target data;

a first quantization error determination module implemented on one or more processing circuits and configured to determine a quantization error corresponding to the target data by

determining a difference between the plurality of pieces of data to be quantized and a plurality of pieces of de-quantized data of the plurality of quantized data; or

determining a difference as a total quantization interval over the plurality of pieces of data to be quantized;

wherein the quantization error is an average of the differences over a sum of the plurality of pieces of data to be quantized; and

an adjusted bit width determination module implemented on one or more processing circuits and configured to, according to the quantization error and an error threshold determined based on an empirical value, adjust a data bit width of each piece of data to be quantized to obtain an adjusted bit width corresponding to each piece of data to be quantized.

11 . The neural network quantization device of claim 10 , wherein the layer to be quantized is a convolution layer, the target data is an input neuron, and the data determination module includes:

a first determination sub-module configured to, in the input neuron of the convolution layer, determine a plurality of pieces of data to be quantized corresponding to a convolution kernel according to a dimension and stride of the convolution kernel, where the dimension of the convolution kernel includes height, width, and a count of channels.

12 . The neural network quantization device of claim 10 , wherein the data determination module includes:

a second determination sub-module configured to determine a plurality of pieces of data to be quantized in the target data of the layer to be quantized according to the dimension of the target data, where the dimension of the target data includes a count of batches, channel, height, and width,

wherein the second determination sub-module includes a determination sub-module based on a count of batches or channel configured to determine one or more batches of data in the target data of the layer to be quantized as one piece of data to be quantized.

13 . The neural network quantization device of claim 10 , wherein the data determination module includes:

a third determination sub-module configured to, according to a real-time processing capability of a device running the neural network, determine a plurality of pieces of data to be quantized in the target data of the layer to be quantized, where a size of each piece of data to be quantized is positively correlated with the real-time processing capability.

14 . The neural network quantization device of claim 10 , further comprising:

a parameter determination sub-module configured to compute the corresponding quantization parameter according to each piece of data to be quantized and a corresponding data bit width;

wherein the parameter determination sub-module includes:

a first point position determination sub-module configured to, when the quantization parameter does not include an offset of each piece of data to be quantized, obtain a first type of point position of each piece of data to be quantized according to a maximum of an absolute value of each piece of data to be quantized and the corresponding data bit width;

a first maximum determination sub-module configured to, when the quantization parameter does not include the offset of each piece of data to be quantized, obtain a maximum of the piece of quantized data according to each piece of data to be quantized and the corresponding data bit width;

a first scaling factor determination sub-module configured to obtain a first type of scaling factor of each piece of data to be quantized according to the maximum of the absolute value of each piece of data to be quantized and the maximum of the piece of quantized data;

a second point position determination sub-module configured to, when the quantization parameter includes the offset of each piece of data to be quantized, obtain a second type of point position of each piece of data to be quantized according to the maximum and a minimum of each piece of data to be quantized and the corresponding data bit width;

a second maximum determination sub-module configured to, when the quantization parameter includes the offset of each piece of data to be quantized, obtain the maximum of the piece of quantized data according to each piece of data to be quantized and the corresponding data bit width; and

a second scaling factor determination sub-module configured to obtain a second type of scaling factor of each piece of data to be quantized according to the maximum and the minimum of each piece of data to be quantized, and the maximum of the piece of quantized data.

15 . The neural network quantization device of 10 , further comprising:

an adjusted quantization parameter determination module configured to update the data bit width corresponding to each piece of data to be quantized to a corresponding adjusted bit width, and compute a corresponding adjusted quantization parameter according to each piece of data to be quantized and the corresponding adjusted bit width to quantize each piece of data to be quantized according to the corresponding adjusted quantization parameter;

a first adjusted quantization error module configured to compute an adjusted quantization error of each piece of data to be quantized according to the data to be quantized and the corresponding adjusted bit width; and

a first adjusting bit width cycle determination module configured to continue to increase the corresponding adjusted bit width according to the adjusted quantization error and the first error threshold until the adjusted quantization error is less than or equal to the first error threshold;

wherein the adjusted bit width determination module includes:

a first adjusted bit width determination sub-module configured to, when the quantization error is greater than a first error threshold, increase the corresponding data bit width to obtain the corresponding adjusted bit width; and

a second adjusted bit width determination sub-module configured to, when the quantization error is less than a second error threshold, decrease the corresponding data bit width to obtain the corresponding adjusted bit width, where the second error threshold is less than the first error threshold.

16 . The neural network quantization device of claim 10 , wherein, during a fine-tuning stage and/or training stage of a neural network operation, the device further includes:

a first data variation range determination module configured to obtain a variation range of the data to be quantized in a current iteration and historical iterations, where the historical iterations are iterations before the current iteration; and

a target iteration interval determination module configured to, according to the variation range of the data to be quantized, determine a target iteration interval corresponding to the data to be quantized to enable the layer to be quantized to update the quantization parameter of the data to be quantized according to the target iteration interval, where the target iteration interval includes at least one iteration.

17 . The neural network quantization device of claim 16 , further comprising:

a first target iteration interval application module configured to, according to the data bit width of the data to be quantized in the current iteration, determine the data bit width of the data to be quantized in the iterations within the target iteration interval to enable the neural network to determine the quantization parameter according to the data bit width of the data to be quantized in the iterations within the target iteration interval; and

a second target iteration interval application module configured to, according to a point position of the data to be quantized in the current iteration, determine the point position of the data to be quantized in the iterations within the target iteration interval, where the point position includes the first type of point position and/or the second type of point position.

18 . An artificial intelligence chip, comprising the neural network quantization device of claim 10 .

19 . A non-transitory computer readable storage medium, on which a computer program instruction is stored, wherein when the computer program instruction is executed by a processor, the neural network quantization method of claim 1 is realized.

Assignments (1)

Priority Claims (2)

Continuity (1)

References Cited (219)