Computationally efficient general matrix-matrix multiplication (GeMM)
A computation unit that comprises (i) a multiplicand vector decomposer that generates a decomposed multiplicand vector which uses a sequence of first and second concatenated multiplicand sub-elements (1 st 2 nd CMCSE) in a lower-precision format (LPF) to represent corresponding ones of multiplicand elements in a multiplicand vector in a higher-precision format (HPF), (ii) a multiplier vector decomposer that generates a decomposed multiplier vector which uses a sequence of first and second concatenated multiplier sub-elements (1 st 2 nd CMLSE) in the LPF to represent corresponding ones of multiplier elements in a multiplier vector in the HPF, (iii) a multiplicand tensor encoder that encodes double reads of the sequence of the 1 st 2 nd CMCSE in a decomposed multiplicand tensor, and (iv) a product vector generator that generates a product vector containing a sequence of first and second concatenated product sub-elements by executing general matrix-matrix multiplication (GeMM) operations between the double reads of the 1 st 2 nd CMCSE and corresponding ones of the 1 st 2 nd CMLSE.
1. A computation unit for general matrix-matrix multiplication (GeMM), comprising:
a multiplicand vector decomposer that traverses a multiplicand vector containing a sequence of multiplicand elements in a higher-precision format and generates a decomposed multiplicand vector that uses a sequence of first and second concatenated multiplicand sub-elements in a lower-precision format to represent corresponding ones of the multiplicand elements;
a multiplier vector decomposer that traverses a multiplier vector containing a sequence of multiplier elements in the higher-precision format and generates a decomposed multiplier vector that uses a sequence of first and second concatenated multiplier sub-elements in the lower-precision format to represent corresponding ones of the multiplier elements;
wherein the higher-precision format has more bits than the lower-precision format;
a multiplicand tensor encoder that traverses the decomposed multiplicand vector and encodes double reads of the sequence of the first and second concatenated multiplicand sub-elements in a decomposed multiplicand tensor; and
a product vector generator that traverses the decomposed multiplicand tensor and the decomposed multiplier vector and generates a product vector containing a sequence of first and second concatenated product sub-elements by executing general matrix-matrix multiplication (GeMM) operations between the double reads of the first and second concatenated multiplicand sub-elements and corresponding ones of the first and second concatenated multiplier sub-elements.
2. The computation unit of claim 1 , wherein the higher-precision format is floating point 32 (FP32).
3. The computation unit of claim 1 , wherein the lower-precision format is Brain floating-point format 16 (bfloat16).
4. The computation unit of claim 1 , wherein the first concatenated multiplicand sub-element in the lower-precision format is generated based on a truncation logic that truncates one or more bits from the corresponding multiplicand element in the higher-precision format to match the lower-precision format.
5. The computation unit of claim 4 , wherein the second concatenated multiplicand sub-element in the lower-precision format is generated based on a residual logic that subtracts, in the higher-precision format, the first concatenated multiplicand sub-element from the corresponding multiplicand element and truncates one or more bits from the result of the subtraction to match the lower-precision format.
6. The computation unit of claim 5 , wherein the second concatenated multiplier sub-element in the lower-precision format is generated based on the residual logic that subtracts, in the higher-precision format, the first concatenated multiplier sub-element from the corresponding multiplier element and truncates one or more bits from the result of the subtraction to match the lower-precision format.
7. The computation unit of claim 1 , wherein the first concatenated multiplier sub-element in the lower-precision format is generated based on the truncation logic that truncates one or more bits from the corresponding multiplier element in the higher-precision format to match the lower-precision format.
8. The computation unit of claim 1 , wherein the sequence of the first and second concatenated product sub-elements is in the higher-precision format.
9. The computation unit of claim 8 , further configured to sum the first and second concatenated product sub-elements and produce a sequence of product elements in the higher-precision format.
10. The computation unit of claim 9 , further configured to convert the product elements from the higher-precision format into the first and second concatenated product sub-elements in the lower-precision format based on the truncation logic and the residual logic.
11. The computation unit of claim 8 , wherein a single instance of the first and second concatenated product sub-elements is produced by using only one GeMM operation, as opposed to four separate GeMM operations, by incorporating two out of three additions of intermediate products within the multiply-accumulate steps of the only one GeMM operation.
12. The computation unit of claim 11 , further configured to save drain overhead by using the only one GeMM operation, as opposed to the four separate GeMM operations.
13. A reconfigurable data processor, comprising:
an array of configurable units; and
a bus system connected to the array of configurable units which communicates data at a processor clock rate, wherein a pattern compute unit (PCU) in the array of configurable units (CGRA) includes a functional unit, the functional unit including:
a multiplicand vector decomposer that traverses a multiplicand vector containing a sequence of multiplicand elements in a higher-precision format and generates a decomposed multiplicand vector that uses a sequence of first and second concatenated multiplicand sub-elements in a lower-precision format to represent corresponding ones of the multiplicand elements;
a multiplier vector decomposer that traverses a multiplier vector containing a sequence of multiplier elements in the higher-precision format and generates a decomposed multiplier vector that uses a sequence of first and second concatenated multiplier sub-elements in the lower-precision format to represent corresponding ones of the multiplier elements;
wherein the higher-precision format has more bits than the lower-precision format;
a multiplicand tensor encoder that traverses the decomposed multiplicand vector and encodes double reads of the sequence of the first and second concatenated multiplicand sub-elements in a decomposed multiplicand tensor; and
a product vector generator that traverses the decomposed multiplicand tensor and the decomposed multiplier vector and generates a product vector containing a sequence of first and second concatenated product sub-elements by executing general matrix-matrix multiplication (GeMM) operations between the double reads of the first and second concatenated multiplicand sub-elements and corresponding ones of the first and second concatenated multiplier sub-elements.
14. The reconfigurable data processor of claim 13 , wherein the higher-precision format is floating point 32 (FP32).
15. The reconfigurable data processor of claim 13 , wherein the lower-precision format is Brain floating-point format 16 (bfloat16).
16. The reconfigurable data processor of claim 13 , wherein the first concatenated multiplicand sub-element in the lower-precision format is generated based on a truncation logic that truncates one or more bits from the corresponding multiplicand element in the higher-precision format to match the lower-precision format.
17. The reconfigurable data processor of claim 16 , wherein the second concatenated multiplicand sub-element in the lower-precision format is generated based on a residual logic that subtracts, in the higher-precision format, the first concatenated multiplicand sub-element from the corresponding multiplicand element and truncates one or more bits from the result of the subtraction to match the lower-precision format.
18. A computer-implemented method, comprising:
traversing a multiplicand vector containing a sequence of multiplicand elements in a higher-precision format and generating a decomposed multiplicand vector that uses a sequence of first and second concatenated multiplicand sub-elements in a lower-precision format to represent corresponding ones of the multiplicand elements;
traversing a multiplier vector containing a sequence of multiplier elements in the higher-precision format and generating a decomposed multiplier vector that uses a sequence of first and second concatenated multiplier sub-elements in the lower-precision format to represent corresponding ones of the multiplier elements;
wherein the higher-precision format has more bits than the lower-precision format;
traversing the decomposed multiplicand vector and encoding double reads of the sequence of the first and second concatenated multiplicand sub-elements in a decomposed multiplicand tensor; and
traversing the decomposed multiplicand tensor and the decomposed multiplier vector and generating a product vector containing a sequence of first and second concatenated product sub-elements by executing general matrix-matrix multiplication (GeMM) operations between the double reads of the first and second concatenated multiplicand sub-elements and corresponding ones of the first and second concatenated multiplier sub-elements.
19. The computer-implemented method of claim 18 , wherein the higher-precision format is floating point 32 (FP32).
20. The computer-implemented method of claim 18 , wherein the lower-precision format is Brain floating-point format 16 (bfloat16).