Methods and apparatuses for high performance and accuracy fixed-point batchnorm implementation

1 . A data processing method, comprising:

receiving, by a hardware chip based on a neural network, floating-point input data over a channel of a standalone floating-point batchnorm layer in the neural network, and converting the floating-point input data into fixed-point input data of the standalone floating-point batchnorm layer;

obtaining, by the hardware chip, fixed-point quantization parameters in each channel based on input data and three floating-point parameters μ i , σ i , ε i in each channel;

converting, by the hardware chip, the standalone floating-point batchnorm layer based on the fixed-point quantization parameters into a fixed-point batchnorm layer for processing the fixed-point input data to generate fixed-point output data; and

mapping, by the hardware chip, the fixed-point batchnorm layer to a fixed-point convolution layer, wherein computation of convolution is done by matrix multiplication executed on a General Matrix Multiplication (GEMM) engine, which is a hardwired unit in the hardware chip;

wherein processing the fixed-point input data to generate the fixed-point output data comprises:

multiplying the fixed-point input data with a first fixed-point quantization parameter to receive a first output;

summing up the first output and a second fixed-point quantization parameter to receive a second output;

right shifting with rounding the second output with an accumulator shift to receive a third output;

clamping the third output to receive a fourth output;

multiplying the fourth output with an output scale to receive a fifth output;

right shifting with rounding the fifth output with an output shift to receive a sixth output; and

clamping the sixth output into the fixed-point output data.

2 . The data processing method according to claim 1 , wherein

for the fixed-point input data in a size of 16-bit, the first fixed-point quantization parameter is in a size comprising S 16 ,

the second fixed-point quantization parameter is in a size comprising S 47 ,

the accumulator shift is in a size comprising U 8 ,

the fourth output is in a size comprising S 32 ,

the output scale is in a size comprising U 16 ,

the output shift is in a size comprising U 8 , and

the fixed-point output data is in a size comprising S 16 or U 16 .

3 . The data processing method according to claim 2 , wherein for the fixed-point input data in the size of 16-bit, a range of the second fixed-point quantization parameter is 30 to 47 bits.

4 . The data processing method according to claim 1 , wherein

for the fixed-point input data in a size of 8-bit, the first fixed-point quantization parameter is in a size comprising S 8 or U 8 ,

the second fixed-point quantization parameter is in a size comprising S 31 ,

the accumulator shift is in a size comprising U 8 ,

the fourth output is in a size comprising S 16 ,

the output scale is in a size comprising U 16 ,

the output shift is in a size comprising U 8 , and

the fixed-point output data is in a size comprising S 8 or U 8 .

5 . The data processing method according to claim 4 , wherein for the fixed-point input data in the size of 8-bit, a range of the second fixed-point quantization parameter is 15 to 31 bits.

6 . The data processing method according to claim 1 , wherein mapping the fixed-point batchnorm layer to the fixed-point convolution layer comprises:

multiplying the fixed-point input data with a filter weight for the channel in the fixed-point batchnorm layer to receive a product; and

summing up the product and a bias for the channel in the fixed-point batchnorm layer.

7 . The data processing method according to claim 1 , wherein the standalone floating-point batchnorm layer comprises a plurality of channels, and the fixed-point quantization parameters are generated separately for each of the plurality of channels.

8 . The data processing method according to claim 1 , wherein the matrix multiplication is executed on a GEMM engine or a Multiply-Accumulate (MAC) operations array.

9 . An apparatus for implementing a neural network, comprising:

one or more processors of a hardware chip based on the neural network; and

a memory configured to store instructions executable by the one or more processors;

wherein the one or more processors, upon execution of the instructions, are configured to:

receive floating-point input data over a channel of a standalone floating-point batchnorm layer in the neural network, and convert the floating-point input data into fixed-point input data of the standalone floating-point batchnorm layer;

obtain fixed-point quantization parameters in each channel based on input data and three floating-point parameters μ i , σ i , ε i in each channel;

convert the standalone floating-point batchnorm layer based on the fixed-point quantization parameters into a fixed-point batchnorm layer in the neural network for processing the fixed-point input data to generate fixed-point output data; and

map the fixed-point batchnorm layer to a fixed-point convolution layer, wherein computation of convolution is done by matrix multiplication executed on a GEMM engine, which is a hardwired unit in the hardware chip;

wherein processing the fixed-point input data to generate the fixed-point output data comprises:

multiplying the fixed-point input data with a first fixed-point quantization parameter to receive a first output;

summing up the first output and a second fixed-point quantization parameter to receive a second output;

right shifting with rounding the second output with an accumulator shift to receive a third output;

clamping the third output to receive a fourth output;

multiplying the fourth output with an output scale to receive a fifth output;

right shifting with rounding the fifth output with an output shift to receive a sixth output; and

clamping the sixth output into the fixed-point output data.

10 . The apparatus of claim 9 , wherein

for the fixed-point input data in a size of 16-bit, the first fixed-point quantization parameter is in a size comprising S 16 ,

the second fixed-point quantization parameter is in a size comprising S 47 ,

the accumulator shift is in a size comprising U 8 ,

the fourth output is in a size comprising S 32 ;

the output scale is in a size comprising U 16 ,

the output shift is in a size comprising U 8 , and

the output data is in a size comprising S 16 or U 16 .

11 . The apparatus of claim 10 , wherein for the fixed-point input data in the size of 16-bit, a range of the second fixed-point quantization parameter is 30 to 47 bits.

12 . The apparatus of claim 9 , wherein

for the fixed-point input data in a size of 8-bit, the first fixed-point quantization parameter is in a size comprising S 8 or U 8 ,

the second fixed-point quantization parameter is in a size comprising S 31 ,

the accumulator shift is in a size comprising U 8 ,

the fourth output is in a size comprising S 16 ,

the output scale is in a size comprising U 16 ,

the third parameter is in a size comprising U 8 , and

the fixed-point output data is in a size comprising S 8 or U 8 .

13 . The apparatus of claim 12 , wherein for the fixed-point input data in the size of 8-bit, a range of the second fixed-point quantization parameter is 15 to 31 bits.

14 . The apparatus of claim 9 , the one or more processors are further configured to:

multiply the fixed-point input data with a filter weight for the channel in the fixed-point batchnorm layer to receive a product; and

sum up the product and a bias for the channel in the fixed-point batchnorm layer.

15 . The apparatus of claim 9 , wherein the standalone floating-point batchnorm layer comprises a plurality of channels, and the fixed-point quantization parameters are generated separately for each of the plurality of channels.

16 . The apparatus of claim 9 , wherein the matrix multiplication is executed on a GEMM engine or a MAC operations array.

17 . A non-transitory computer readable storage medium, comprising instructions stored therein to implement a neural network, wherein, upon execution of the instructions by one or more processors of a hardware chip based on the neural network, the instructions cause the one or more processors to perform acts comprising:

receiving floating-point input data over a channel of a standalone floating-point batchnorm layer in the neural network, and converting the floating-point input data into fixed-point input data of the standalone floating-point batchnorm layer;

obtaining fixed-point quantization parameters in each channel based on input data and three floating-point parameters μ i , σ i , ε i in each channel;

converting the standalone floating-point batchnorm layer based on the fixed-point quantization parameters into a fixed-point batchnorm layer in the neural network for processing the fixed-point input data to generate fixed-point output data; and

mapping the fixed-point batchnorm layer to a fixed-point convolution layer, wherein computation of convolution is done by matrix multiplication that executed on a GEMM engine, which is a hardwired unit in the hardware chip;

wherein processing the fixed-point input data to generate the fixed-point output data comprises:

multiplying the fixed-point input data with a first fixed-point quantization parameter to receive a first output;

summing up the first output and a second fixed-point quantization parameter to receive a second output;

right shifting with rounding the second output with an accumulator shift to receive a third output;

clamping the third output to receive a fourth output;

multiplying the fourth output with an output scale to receive a fifth output;

right shifting with rounding the fifth output with an output shift to receive a sixth output; and

clamping the sixth output into the fixed-point output data.

18 . The non-transitory computer readable storage medium of claim 17 , wherein

for the fixed-point input data in a size of 16-bit, the first fixed-point quantization parameter is in a size comprising S 16 ,

the second fixed-point quantization parameter is in a size comprising S 47 ,

the accumulator shift is in a size comprising U 8 ,

the fourth output is in a size comprising S 32 ,

the output scale is in a size comprising U 16 ,

the output shift is in a size comprising U 8 , and

the fixed-point output data is in a size comprising S 16 or U 16 .

19 . The non-transitory computer readable storage medium of claim 18 , wherein for the fixed-point input data in the size of 16-bit, a range of the second fixed-point quantization parameter is 30 to 47 bits.

20 . The non-transitory computer readable storage medium of claim 17 , wherein

for the fixed-point input data in a size of 8-bit, the first fixed-point quantization parameter is in a size comprising S 8 or U 8 ,

the second fixed-point quantization parameter is in a size comprising S 31 ,

the accumulator shift is in a size comprising U 8 ,

the fourth output is in a size comprising S 16 ,

the output scale is in a size comprising U 16 ,

the third parameter is in a size comprising U 8 ,

and the fixed-point output data is in a size comprising S 8 or U 8 .

21 . The non-transitory computer readable storage medium of claim 20 , wherein for the fixed-point input data in the size of 8-bit, a range of the second fixed-point quantization parameter is 15 to 31 bits.

22 . The non-transitory computer readable storage medium of claim 17 , wherein mapping the fixed-point batchnorm layer to the fixed-point convolution layer comprises:

multiplying the fixed-point input data with a filter weight for the channel in the fixed-point batchnorm layer to receive a product; and

summing up the product and a bias for the channel in the fixed-point batchnorm layer.

23 . The non-transitory computer readable storage medium of claim 17 , wherein the standalone floating-point batchnorm layer comprises a plurality of channels, and the fixed-point quantization parameters are generated separately for each of the plurality of channels.

24 . The non-transitory computer readable storage medium of claim 17 , wherein the matrix multiplication is executed on GEMM engine or a MAC operations array.