Accelerated quantized multiply-and-add operations
Disclosed herein are techniques for accelerating convolution operations or other matrix multiplications in applications such as neural network. A computer-implemented method includes receiving low-precision inputs for a convolution operation from a storage device, and subtracting a low-precision value representing a high-precision zero value from the low-precision inputs to generate difference values, where the low-precision inputs are asymmetrically quantized from high-precision inputs. The method also includes performing multiplication and summation operations on the difference values to generate a sum of products, and generating a high-precision output by scaling the sum of products with a scaling factor.
1. A computer-implemented method, comprising:
receiving floating point inputs to a convolution layer of an artificial neural network;
quantizing the floating point inputs linearly into n-bit integer inputs, wherein:
n is a multiple of 8;
a minimum value of the floating point inputs is quantized as integer zero;
a maximum value of the floating point inputs is quantized as 2 n −1; and
a zero value of the floating point inputs is quantized as an integer representing the zero value of the floating point inputs with no quantization error;
storing the integer inputs in a memory;
reading, by the convolution layer, the integer inputs from the memory;
creating difference values by subtracting, by a subtraction engine, the integer representing the zero value of the floating point inputs from the integer inputs such that the zero value of the floating point inputs is represented by an integer value zero;
performing, by the convolution layer, one set of (n+1)-bit integer multiplication and summation operations on the difference values to generate a sum of products; and
generating, by an activation layer, a pooling layer, or an output buffer of the artificial neural network, a floating point output by scaling the sum of products with a floating point scaling factor.
2. The computer-implemented method of claim 1 , wherein n is equal to 8.
3. The computer-implemented method of claim 1 , wherein the minimum value and the maximum value of the floating point inputs are not symmetrical with respect to the zero value of the floating point inputs.
4. The computer-implemented method of claim 1 , wherein the floating point output represents a convolution result of the floating point inputs.
5. The computer-implemented method of claim 1 , wherein the floating point scaling factor is a first floating point scaling factor; and
wherein the first floating point scaling factor comprises a multiplication product of second floating point scaling factors, each second floating point scaling factor being associated with the quantization of one of the floating point inputs into the respective n-bit integer input.
6. An apparatus comprising:
a storage device configured to store low-precision inputs for a convolution operation, wherein the low-precision inputs are generated from high-precision inputs using asymmetrical quantization;
an array of subtraction engines configured to receive the low-precision inputs from the storage device and subtract a low-precision value representing a high-precision zero value from the low-precision inputs, thereby creating difference values;
a processing element (PE) array configured to generate a sum of products using the difference values, the PE array including a plurality of rows of processing elements, each row of processing elements configured to receive difference values from a corresponding subtraction engine; and
a processing engine configured to scale the sum of products by a scaling factor to generate a high-precision output.
7. The apparatus of claim 6 , wherein:
the low-precision inputs are stored in the storage device in an n-bit format; and
n is a multiple of 8.
8. The apparatus of claim 7 , wherein:
the high-precision inputs comprise 32-bit floating point inputs;
the low-precision inputs comprise 8-bit unsigned integers; and
the difference values comprise 9-bit signed integers.
9. The apparatus of claim 6 , wherein, in the asymmetrical quantization:
a minimum value of the high-precision inputs is quantized as a zero value in the low-precision inputs;
a maximum value of the high-precision inputs is quantized as a maximum value in the low-precision inputs; and
a zero value of the high-precision inputs is quantized as the low-precision value representing the high-precision zero value with no quantization error.
10. The apparatus of claim 6 , wherein:
the high-precision inputs comprise weights and input data elements for a layer of an artificial neural network; and
the weights and the input data elements are quantized independently.
11. The apparatus of claim 10 , wherein the array of subtraction engines is configured to subtract from low-precision inputs corresponding to both the weights and the input data elements.
12. The apparatus of claim 6 , wherein the apparatus comprises a field-programmable gate array (FPGA), a system-on-chip (SoC), an application specific integrated circuit (ASIC), a programmable array logic (PAL), or a complex programmable logic device (CPLD).
13. The apparatus of claim 6 , wherein the processing element array is configured to perform one set of multiplications and summations using the difference values to generate the sum of products.
14. The apparatus of claim 6 , wherein the processing engine comprises an activation engine, a pooling engine, or an output buffer of an artificial neural network.
15. The apparatus of claim 6 , wherein the sum of products is generated by the PE array in a single pass.
16. A computer-implemented method, comprising:
receiving, from a storage device, low-precision inputs for a convolution operation, wherein the low-precision inputs are generated from high-precision inputs using asymmetrical quantization;
subtracting, by a subtraction engine, a low-precision value representing a high-precision zero value from the low-precision inputs to create difference values;
performing, by a processing engine, multiplication and summation operations on the difference values to generate a sum of products; and
generating, by a post-processing engine, a high-precision output by scaling the sum of products with a scaling factor.
17. The computer-implemented method of claim 16 , wherein:
the high-precision inputs comprise 32-bit floating point inputs;
the low-precision inputs comprise 8-bit unsigned integers; and
the difference values comprise 9-bit signed integers.
18. The computer-implemented method of claim 16 , wherein, in the asymmetrical quantization:
a minimum value of the high-precision inputs is quantized as a zero value in the low-precision inputs;
a maximum value of the high-precision inputs is quantized as a maximum value in the low-precision inputs; and
a zero value of the high-precision inputs is quantized as the low-precision value representing the high-precision zero value with no quantization error.
19. The computer-implemented method of claim 18 , wherein the minimum value and the maximum value of the high-precision inputs are not symmetrical with respect to the zero value of the high-precision inputs.
20. The computer-implemented method of claim 16 , wherein the high-precision inputs comprise weights or input data elements for a layer of an artificial neural network.
21. The computer-implemented method of claim 16 , wherein the processing engine is configured to perform low-precision multiplication and summation operations rather than high-precision multiplication and summation operations.
22. The computer-implemented method of claim 16 , wherein the multiplication and summation operations are performed by a processing element array.
23. The computer-implemented method of claim 16 , wherein scaling the sum of products with the scaling factor comprises scaling the sum of products using an activation engine, a pooling engine, or an output buffer of an artificial neural network.