Bit-parallel vector composability for neural acceleration
View Patent ↗Methods, apparatus and systems that relate to hardware accelerators of artificial neural network (ANN) performance that significantly reduce the energy and area costs associated with performing vector dot-product operations in the ANN training and inference tasks. Specifically, the methods, apparatus and systems reduce the cost of bit-level flexibility stemming from aggregation logic by amortizing related costs across vector elements and reducing complexity of the cooperating narrower bitwidth units.
1. An apparatus for performing an energy-efficient dot-product operation between two input vectors, comprising:
a plurality of vector computation engines configurable into multiple groups of vector computation engines, wherein each vector computation engine from the plurality of vector computation engines comprises:
an array of multipliers connected through one or more add units and configured to
generate an output of the vector computation engine based on a dot-product
operation on a subset of bits of the two input vectors;
a plurality of shifters configured to shift the outputs of the vector computation engines based on a significance position, within the two input vectors, of a sub-vector associated with the outputs; and
an aggregator coupled to the plurality of shifters and configured to generate a scalar output for the energy-efficient dot-product operation based on aggregating the shifted outputs, wherein a number of the vector computation engines is based, in part, on bitwidths of datatypes of elements of the two input vectors,
wherein the multiple groups are operable according to levels including at least one level in which outputs generated by each of the multiple groups are aggregated to produce a final scalar result for the energy-efficient dot-product operation.
2. The apparatus of claim 1 , wherein the plurality of vector computation engines is configured to operate in parallel.
3. The apparatus of claim 1 , wherein one of the two vectors is a vector of weights of a neural network.
4. The apparatus of claim 1 , wherein the number of the vector computation engines is further based on a size of the subset of bits of the two input vectors.
5. The apparatus of claim 1 , wherein the number of the vector computation engines is based on a size of the subset of bits of the two input vectors and a maximum bitwidth of datatypes of elements of the two input vectors.
6. The apparatus of claim 1 , wherein each multiplier in the array of multipliers is configured to perform a 2-bit by 2-bit multiplication.
7. The apparatus of claim 6 , wherein the number of multipliers in the array of multipliers is 16.
8. The apparatus of claim 1 , wherein each multiplier in the array of multipliers is configured to perform a 4-bit by 4-bit multiplication.
9. The apparatus of claim 1 , wherein the plurality of vector computation engines is configurable into multiple groups of vector computation engines according to bitwidths of elements of the two input vectors.
10. The apparatus of claim 1 , wherein the plurality of shifters is configurable according to bitwidths of elements of the two input vectors.
11. The apparatus of claim 1 , wherein the multipliers of the array of multipliers and add units are configurable according to bitwidths of elements of the two input vectors.
12. The apparatus of claim 3 , comprising a memory configured to store the vector of weights of the neural network.
13. An apparatus for performing an energy-efficient dot-product operation between a first vector and a second vector, comprising:
a plurality of vector computation engines configurable into multiple groups of vector computation engines, the plurality of vector computation engines including,
a first vector computation engine comprising a first group of multipliers and add units, wherein the first vector computation engine, with the first group of multipliers and add units, is configured to produce a first dot-product of a third vector and a fourth vector, wherein each element of the third vector includes a subset of bits of a corresponding element of the first vector and each element of the fourth vector includes a subset of bits of a corresponding element of the second vector, and
a second vector computation engine comprising a second group of multipliers and add units, wherein the second vector computation engine, with the second group of multipliers and add units, is configured to produce a second dot-product of a fifth vector and a sixth vector, wherein each element of the fifth vector includes a subset of bits of a corresponding element of the first vector and each element of the sixth vector includes a subset of bits of a corresponding element of the second vector;
a first bit shifter configured to bit-shift the first dot-product by a first number of bits;
a second bit shifter configured to bit-shift the second dot-product by a second number of bits; and
an aggregator configured to aggregate the bit-shifted first dot-product and the bit-shifted second dot-product,
wherein the apparatus is configured to generate at least one of the first dot-product or the bit-shifted first dot-product in parallel with at least one of the second dot-product or the bit-shifted second dot-product,
wherein one of the first vector or the second vector is a vector of inputs of a layer of a neural network, wherein results of the first vector computation engine and the second vector computation engine are combined according to a bitwidth of the layer of the neural network,
wherein the multiple groups are operable according to levels including at least one level in which outputs generated by each of the multiple groups are aggregated to produce a final scalar result for the energy-efficient dot-product operation.
14. A method of performing an energy-efficient dot-product operation between two vectors, comprising:
generating partial dot-product outputs of a plurality of vector computation engines configurable into multiple groups of vector computation engines, wherein each partial dot-product output is based on performing a dot-product operation on a subset of bits of the two vectors using an array of multipliers connected through one or more add units;
shifting the partial dot-product outputs of the vector computation engines belonging to the multiple groups using bit shifters; and
aggregating the shifted partial dot-product outputs of each of the multiple groups using an aggregator to produce a scalar output for the energy-efficient dot-product operation,
wherein the generating partial dot-product outputs is performed in a parallel manner, wherein a number of the vector computation engines is based, in part, on bitwidths of datatypes of elements of the two vectors,
wherein the multiple groups are operable according to levels including at least one level in which outputs generated by each of the multiple groups are aggregated to produce a final scalar result for the energy-efficient dot-product operation.
15. The method of claim 14 , wherein one of the two vectors is a vector of weights of a neural network.
16. The method of claim 14 , wherein the number of the partial dot-product outputs is based on at least one of:
a size of the subset of bits of the two vectors and bitwidths of datatypes of components of the two vectors, or
a size of the subset of bits of the two vectors and a maximum bitwidth of datatypes of components of the two vectors.
17. The method of claim 14 , wherein each multiplier is configured to perform a 2-bit by 2-bit multiplication.
18. The method of claim 14 , wherein each multiplier is configured to perform a 4-bit by 4-bit multiplication.
19. The method of claim 14 , comprising configuring the shifter elements based on bitwidths of components of the two vectors.
20. The method of claim 14 , comprising configuring the multiplies and add units based on bitwidths of components of the two vectors.