Method and apparatus for keeping statistical inference accuracy with 8-bit Winograd convolution
A method and apparatus for keeping statistical inference accuracy with 8-bit winograd convolution. A calibration dataset and a pretrained CNN comprising 32-bit floating point weight values may be sampled to generate an input activation tensor and a weight tensor. A transformed input activation tensor may be generated by multiplying the input activation tensor and an input matrix to generate a transformed input activation tensor. A transformed weight tensor may be generated by multiplying the weight tensor and a weight matrix. A scale factor may be computed for each transformed tensor. An 8-bit CNN model including the scale factors may be generated.
1. An apparatus, comprising:
a processor; and
a memory storing instructions which when executed by the processor cause the processor to:
input an image into a neural network,
obtain an input activation tensor of a convolutional layer in the neural network based on the image and obtain a weight tensor of the convolutional layer, wherein the convolutional layer has a convolution of a floating-point data type, wherein values in the input activation tensor and the weight tensor have the floating-point data type,
generate a transformed input activation tensor by transforming the input activation tensor with a first constant matrix, one or more values in the transformed input activation tensor having an integer data type,
generate a transformed weight tensor by transforming the weight tensor with a second constant matrix, one or more values in the transformed weight tensor having the integer data type,
compute scale factors from the transformed input activation tensor and the transformed weight tensor,
perform a convolution of the integer data type, in lieu of the convolution of the floating-point type, on the transformed input activation tensor and the transformed weight tensor, wherein the convolution of the integer data type is performed by an algorithm logic unit of the floating-point data type based on the scaled factors, and
generate a classification label of the image based on a result of the convolution of the integer data type, the classification label indicating recognition of an object in the image.
2. The apparatus of claim 1 , wherein the algorithm logic unit is implemented in a hardware accelerator configured to:
execute the convolutional layer by performing an 8-bit Coppersmith-Winograd (CW) convolution operation on the transformed input activation tensor and the transformed weight tensor.
3. The apparatus of claim 1 , wherein the memory stores instructions for one or more quantization functions to: (i) quantize 32-bit floating point weight values to 8-bit integers, (ii) dequantize 8-bit integers to 32-bit floating point weight values, (iii) perform matrix addition operations, and (iv) perform matrix multiplication operations.
4. The apparatus of claim 3 , wherein the quantization function to quantize 32-bit floating point weight values to 8-bit integers comprises:
Q ( r,q,p )= Q p ( r,q )= Q p,q ( r )=( z,q ) z =max(min(Round( qr ),2 p −1),−2 p ),
where r∈R n ,q∈R,p∈N + ,z∈Z n ,Round: R n →Z n ,
where Q comprises the quantization function, r comprises an n-dimensional rational tensor, and z comprises an n-dimensional rational tensor with a scale factor q and bit-precision p.
5. The apparatus of claim 4 , wherein the quantization function to dequantize 8-bit integers to 32-bit floating point weight values comprises:
D
(
z
,
q
)
=
D
q
(
z
)
=
z
q
=
r
′
≈
r
.
6. The apparatus of claim 5 , wherein the quantization function to perform matrix addition operations comprises:
( z 1 ,q 1 )+( z 2 ,q 2 )= Q p ( D q1 ( z 1 )+ D q2 ( z 2 ),min( q 1 ,q 2 )).
7. The apparatus of claim 6 , wherein the quantization function to perform matrix multiplication operations comprises:
( z 1 ,q 1 )×( z 2 ,q 2 )=( z 1 ×z 2 ,q 1 q 2 ).
8. The apparatus of claim 7 , wherein the quantization functions further comprise a function to define a scale factor for direct Coppersmith-Winograd (CW) convolution using the following equation:
2
p
-
1
max
,
where p=7 for the weight tensor, where p=8 for the input activation tensor, where max comprises a maximum value in each respective tensor.
9. The apparatus of claim 1 , wherein the transformed input activation tensor is computed based on the following equation:
x a =B T x b ,
where x a comprises a value of the transformed input activation tensor, the first constant matrix comprises B T , and x b comprises a value of the input activation tensor.
10. The apparatus of claim 9 , wherein a scale factor for the transformed input activation tensor is computed based on the following equation:
q
x
a
=
q
x
b
×
max
x
b
max
x
a
,
where q x a comprises the scale factor for the transformed input activation tensor, max x b corresponds to a maximum value in the input activation tensor, max x a corresponds to a maximum value transformed input activation tensor, and q x b comprises a quantized value of the input activation tensor.
11. The apparatus of claim 1 , wherein the transformed weight tensor is computed based on the following equation:
w a =Gw b G T ,
where w a comprises a value of the transformed weight tensor, the second constant matrix comprises G, G T is a transpose of the second constant matrix, and w b comprises a value of the weight tensor.
12. The apparatus of claim 11 , wherein a scale factor for the transformed weight tensor is computed based on the following equation:
q
w
a
=
q
w
b
×
max
w
b
max
w
a
,
where q w a comprises the scale factor for the transformed weight tensor, max w b corresponds to a maximum value in the weight tensor, max w a corresponds to a maximum value transformed weight tensor, and q w b comprises a quantized value of the weight tensor.
13. A method, comprising:
inputting an image into a neural network;
obtaining an input activation tensor of a convolutional layer in the neural network based on the image and obtaining a weight tensor of the convolutional layer, wherein the convolutional layer has a convolution of a floating-point data type, wherein values in the input activation tensor and the weight tensor have the floating-point data type;
generating a transformed input activation tensor by transforming the input activation tensor with a first constant matrix, one or more values in the transformed input activation tensor having an integer data type;
generating a transformed weight tensor by transforming the weight tensor with a second constant matrix, one or more values in the transformed weight tensor having the integer data type;
computing scale factors from the transformed input activation tensor and the transformed weight tensor;
performing a convolution of the integer data type, in lieu of the convolution of the floating-point type, on the transformed input activation tensor and the transformed weight tensor, wherein the convolution of the integer data type is performed by an algorithm logic unit of the floating-point data type based on the scaled factors; and
generating a classification label of the image based on a result of the convolution of the integer data type, the classification label indicating recognition of an object in the image.
14. The method of claim 13 , wherein the convolution of the integer data type is an 8-bit Coppersmith-Winograd (CW) convolution.
15. The method of claim 13 , further comprising performing one or more quantization functions to: (i) quantize 32-bit floating point weight values to 8-bit integers, (ii) dequantize 8-bit integers to 32-bit floating point weight values, (iii) perform matrix addition operations, and (iv) perform matrix multiplication operations.
16. The method of claim 15 , wherein the quantization function to quantize 32-bit floating point weight values to 8-bit integers comprises:
Q ( r,q,p )= Q p ( r,q )= Q p,q ( r )=( z,q ) z =max(min(Round( qr ),2 p −1),−2 p ),
where r∈R n ,q∈R,p∈N + ,z∈Z n ,Round: R n →Z n ,
where Q comprises the quantization function, r comprises an n-dimensional rational tensor, and z comprises an n-dimensional rational tensor with a scale factor q and bit-precision p.
17. The method of claim 16 , wherein the quantization function to dequantize 8-bit integers to 32-bit floating point weight values comprises:
D
(
z
,
q
)
=
D
q
(
z
)
=
z
q
=
r
′
≈
r
.
18. The method of claim 17 , wherein the quantization function to perform matrix addition operations comprises:
( z 1 ,q 1 )+( z 2 ,q 2 )= Q p ( D q1 ( z 1 )+ D q2 ( z 2 ),min( q 1 ,q 2 )).
19. A non-transitory computer-readable storage medium comprising instructions that when executed by a computing device, cause the computing device to:
input an image into a neural network;
obtain an input activation tensor of a convolutional layer in the neural network based on the image and obtain a weight tensor of the convolutional layer, wherein the convolutional layer has a convolution of a floating-point data type, wherein values in the input activation tensor and the weight tensor have the floating-point data type;
generate a transformed input activation tensor by transforming the input activation tensor with a first constant matrix, one or more values in the transformed input activation tensor having an integer data type;
generate a transformed weight tensor by transforming the weight tensor with a second constant matrix, one or more values in the transformed weight tensor having the integer data type;
compute scale factors from the transformed input activation tensor and the transformed weight tensor;
perform a convolution of the integer data type, in lieu of the convolution of the floating-point type, on the transformed input activation tensor and the transformed weight tensor, wherein the convolution of the integer data type is performed by an algorithm logic unit of the floating-point data type based on the scaled factors; and
generate a classification label of the image based on a result of the convolution of the integer data type, the classification label indicating recognition of an object in the image.
20. The non-transitory computer-readable storage medium of claim 19 , further comprising instructions executable by the computing device to cause the computing device to:
execute the convolutional layer by performing an 8-bit Coppersmith-Winograd (CW) convolution operation on the transformed input activation tensor and the transformed weight tensor.
21. The non-transitory computer-readable storage medium of claim 19 , further comprising instructions executable by the computing device to cause the computing device to perform one or more quantization functions executable by the computing device to cause the computing device to: (i) quantize 32-bit floating point weight values to 8-bit integers, (ii) dequantize 8-bit integers to 32-bit floating point weight values, (iii) perform matrix addition operations, and (iv) perform matrix multiplication operations.
22. The non-transitory computer-readable storage medium of claim 21 , wherein the quantization function to quantize 32-bit floating point weight values to 8-bit integers comprises:
Q ( r,q,p )= Q p ( r,q )= Q p,q ( r )=( z,q ) z =max(min(Round( qr ),2 p −1),−2 p ),
where r∈R n ,q∈R,p∈N + ,z∈Z n ,Round: R n →Z n ,
where Q comprises the quantization function, r comprises an n-dimensional rational tensor, and z comprises an n-dimensional rational tensor with a scale factor q and bit-precision p.
23. The non-transitory computer-readable storage medium of claim 22 , wherein the quantization function to dequantize 8-bit integers to 32-bit floating point weight values comprises:
D ( z,q )= D q ( z )= z/q=r′≈r.
24. The non-transitory computer-readable storage medium of claim 23 , wherein the quantization function to perform matrix addition operations comprises:
( z 1 , q 1 )+( z 2 ,q 2 )= Q p ( D q1 ( z 1 )+ D q2 ( z 2 ),min( q 1 ,q 2 )).
25. The non-transitory computer-readable storage medium of claim 24 , wherein the quantization function to perform matrix multiplication operations comprises:
( z 1 ,q 1 )×( z 2 ,q 2 )=( z 1 ×z 2 ,q 1 q 2 ).