Flexible compute engine microarchitecture
View Patent ↗A compute engine is described. The compute engine includes compute-in-memory (CIM) modules and may include an input buffer coupled with the CIM modules. The input buffer stores a vector. The CIM modules store weights corresponding to a matrix and perform a vector-matrix multiplication (VMM) for the matrix and the vector. The CIM modules further include storage cells and vector multiplication units (VMUs) coupled with the storage cell and, if present, the input buffer. The storage cells store the weights. The VMUs multiply, with the vector, at least a portion of each weight of a portion of the plurality of weights corresponding to a portion of the matrix. A set of VMUs performs multiplications for a first weight length and a second weight length different from the first weight length such that each VMU of the set performs multiplications for both the first weight length and the second weight length.
1. A compute engine, comprising:
an input buffer configured to store a vector, the vector including a vector sign; and
a plurality of compute-in-memory (CIM) hardware modules coupled with the input buffer, configured to store a plurality of weights corresponding to a matrix, and configured to perform a vector-matrix multiplication (VMM) for the matrix and the vector, each of the plurality of weights including a weight sign, the plurality of CIM hardware modules further including:
a plurality of storage cells for storing the plurality of weights; and
a plurality of vector multiplication units (VMUs) coupled with the plurality of storage cells and the input buffer, each VMU of the plurality of VMUs configured to multiply, with the vector, at least a portion of a weight of a portion of the plurality of weights;
wherein a set of VMUs of the plurality of VMUs are configured to perform multiplications for a first weight length and a second weight length different from the first weight length such that each VMU of the set of VMUs performs the multiplications for both the first weight length and the second weight length, an indication of the vector sign being propagated to each VMU of the set of VMUs, the weight sign being propagated to a portion of the VMUs in the set of VMUs for the second weight length; and
wherein each VMU in the portion of the VMUs is configured to extend the weight sign for a remaining portion of the VMU for the second weight length.
2. The compute engine of claim 1 , wherein the set of VMUs includes a first VMU and a second VMU, the first VMU and the second VMU each being configured to perform the multiplications for the first weight length, the second weight length being twice the first weight length.
3. The compute engine of claim 1 , further comprising:
a combiner coupled with the set of VMUs, the combiner configured to combine a product of the at least the portion of each weight and an element of the vector for the second weight length.
4. The compute engine of claim 3 , wherein the combiner shifts the product for a first VMU of the set of VMUs by the first weight length relative to the product for a second VMU of the set of VMUs.
5. The compute engine of claim 1 , wherein each element of the vector in the input buffer is serialized for the multiplications.
6. The compute engine of claim 1 , wherein the first weight length corresponds to more significant bits of a weight and the second weight length corresponds to a combination of the more significant bits of the weight and less significant bits of the weight, the set of VMUs including a first VMU for the more significant bits of the weight and at least a second VMU for the less significant bits of the weight.
7. The compute engine of claim 1 , wherein the compute engine has a first output precision for the first weight length and a second output precision for the second weight length, the second output precision being a multiple of the first output precision.
8. The compute engine of claim 1 , wherein each VMU in the set of VMUs determines the multiplications using a lookup table.
9. The compute engine of claim 1 , wherein the input buffer is configured to store non-vector data; and wherein the plurality of CIM hardware modules are configured to multiply the non-vector data with the matrix.
10. The compute engine of claim 1 , wherein the plurality of VMUs are configured to perform multiplications for the first weight length only or the second weight length only for either a plurality of matrices or a plurality of vectors.
11. An accelerator, comprising:
a processor, and
a plurality of compute engines coupled with the processor, each of the plurality of compute engines including an input buffer configured to store a vector and a plurality of compute-in-memory (CIM) hardware modules, coupled with the input buffer, configured to store a plurality of weights corresponding to a matrix, and configured to perform a vector-matrix multiplication (VMM) for the matrix and the vector, the vector including a vector sign, each of the plurality of weights including a weight sign, the plurality of CIM hardware modules further including:
a plurality of storage cells for storing the plurality of weights;
a plurality of vector multiplication units (VMUs) coupled with the plurality of storage cells and the input buffer, each VMU of the plurality of VMUs configured to multiply, with the vector, at least a portion of a weight of a portion of the plurality of weights corresponding to a portion of the matrix;
wherein a set of VMUs of the plurality of VMUs are configured to perform multiplications for a first weight length and a second weight length different from the first weight length such that each VMU of the set of VMUs performs the multiplications for both the first weight length and the second weight length, an indication of the vector sign being propagated to each VMU of the set of VMUs, the weight sign being propagated to a portion of the VMUs in the set of VMUs for the second weight length; and
wherein each VMU in the portion of the VMUs is configured to extend the weight sign for a remaining portion of the VMU for the second weight length.
12. A method, comprising:
providing a vector to a compute engine including a plurality of compute-in-memory (CIM) hardware modules configured to store a plurality of weights corresponding to a matrix and configured to perform a vector-matrix multiplication (VMM) for the matrix, the plurality of CIM hardware modules further including a plurality of storage cells and a plurality of vector matrix multiplication units (VMUs), the plurality of storage cells storing the plurality of weights, the plurality of VMUs coupled with the plurality of storage cells, each VMU of the plurality of VMUs configured to multiply, with the vector, at least a portion of each weight of a portion of the plurality of weights corresponding to a portion of the matrix, the vector including a vector sign and each of the plurality of weights including a weight sign; and
performing a VMM of the vector and the matrix using the plurality of CIM hardware modules, a set of VMUs of the plurality of VMUs configured to perform multiplications for a first weight length and a second weight length different from the first weight length such that each VMU of the set of VMUs is capable of performing the multiplications for both the first weight length and the second weight length, the performing the VMM further including:
propagating an indication of the vector sign to each VMU of the set of VMUs;
propagating the weight sign to a portion of the VMUs in the set of VMUs for the second weight length; and
in each VMU of the portion of the VMUs, extending the weight sign for a remaining portion of the VMU for the second weight length.
13. The method of claim 12 , wherein the set of VMUs includes a first VMU and a second VMU, the first VMU and the second VMU each being configured to perform the multiplications for the first weight length, the second weight length being twice the first weight length.
14. The method of claim 12 , further comprising:
combining a product of the portion of each weight and an element of the vector for the second weight length.
15. The method of claim 14 , wherein the combining further includes:
shifting the product for a first VMU of the set of VMUs by the first weight length relative to the product for a second VMU of the set of VMUs.
16. The method of claim 12 , wherein the providing the vector further includes:
serializing each element of the vector.