Matrix multiplication engine using pipelining
Techniques for data manipulation using a matrix multiplication engine using pipelining are disclosed. A first and a second matrix are obtained for matrix multiplication. A first matrix multiply-accumulate (MAC) unit is configured, where a first matrix element and a second matrix element are presented to the MAC unit on a first cycle. A second MAC unit is configured in pipelined fashion, where the first element of the first matrix and a second element of the second matrix are presented to the second MAC unit on a second cycle, and where a second element of the first matrix and the first element of the second matrix are presented to the first MAC unit on the second cycle. Additional MAC units are further configured within the processor in pipelined fashion. Multiply-accumulate operations are executed in pipelined fashion on each of n MAC units over additional k sets of m cycles.
1. A processor-implemented method for data manipulation comprising:
obtaining a first matrix with dimensions m×k and a second matrix with dimensions k×n for matrix multiplication within a processor;
configuring a first matrix multiply-accumulate (MAC) unit within the processor, wherein the first MAC unit accepts input elements of the first matrix and the second matrix, and wherein a first element of the first matrix and a first element of the second matrix are presented to the first MAC unit on a first cycle;
configuring a second MAC unit within the processor in pipelined fashion, wherein the second MAC unit accepts input elements of the first matrix and the second matrix, wherein the first element of the first matrix and a second element of the second matrix are presented to the second MAC unit on a second cycle, and wherein a second element of the first matrix and the first element of the second matrix are presented to the first MAC unit on the second cycle;
further configuring one or more additional MAC units within the processor in pipelined fashion; and
executing multiply-accumulate operations in pipelined fashion.
2. The method of claim 1 wherein the configuring is accomplished such that a total of n MAC units are configured, wherein an i th MAC unit accepts input elements of the first matrix and the second matrix, wherein i is an integer in a range of (3≤i≤n), and wherein the first element of the first matrix and an i th element of the second matrix are presented to the i th MAC unit on an (i+2) th cycle.
3. The method of claim 2 wherein the executing includes executing multiply-accumulate operations in pipelined fashion on each of the n MAC units for a total of (m×k) cycles, wherein additional m elements from the first matrix are presented over additional k sets of m cycles.
4. The method of claim 1 wherein the executing is accomplished using time-division multiplexing.
5. The method of claim 1 further comprising outputting results of the matrix multiplication to a storage element, wherein the outputting takes an additional (m×k) cycle.
6. The method of claim 1 further comprising presenting a first element from a third matrix and a first element from a fourth matrix to the first MAC unit after (m×k) cycles.
7. The method of claim 6 wherein a second dimension of the fourth matrix is the same as a second dimension of the second matrix.
8. The method of claim 1 wherein the matrices comprise subsections of an o-dimensional tensor, wherein o is greater than 2.
9. The method of claim 1 wherein each MAC unit is configured to have an accumulator depth of m.
10. The method of claim 1 wherein each matrix multiply inner product takes k cycles to complete.
11. The method of claim 1 further comprising pipelining input elements to the MAC units through two input registers.
12. The method of claim 1 wherein the input elements of the first matrix comprise an 8×8 submatrix.
13. The method of claim 12 wherein the input elements of the second matrix comprise an 8×8 submatrix.
14. The method of claim 1 wherein performing n MAC operations in parallel reduces an amount of time taken to perform the n MAC operations from an order of magnitude of n 3 to an order of magnitude of n 2 .
15. The method of claim 1 further comprising adding one or more idle or no operation (NOP) cycles after completion of a matrix multiply operation before starting a next matrix multiply operation.
16. The method of claim 15 wherein the number of idle or NOP cycles is no less than (m×n)−(m×(k−1))−n.
17. The method of claim 15 wherein the adding idle or NOP cycles facilitates pipeline draining.
18. The method of claim 15 wherein the idle or NOP cycles are interpreted when neither an input element of the first matrix nor an input element of the second matrix is valid during a same cycle.
19. The method of claim 1 wherein the MAC units are used to implement a data flow graph.
20. The method of claim 1 wherein the input elements of the first matrix are taken from a row of the first matrix.
21. The method of claim 1 wherein the input elements of the second matrix are taken from a column of the second matrix.
22. A computer program product embodied in a non-transitory computer readable medium for data manipulation, the computer program product comprising code which causes one or more processors to perform operations of:
obtaining a first matrix with dimensions m×k and a second matrix with dimensions k×n for matrix multiplication within a processor;
configuring a first matrix multiply-accumulate (MAC) unit within the processor, wherein the first MAC unit accepts input elements of the first matrix and the second matrix, and wherein a first element of the first matrix and a first element of the second matrix are presented to the first MAC unit on a first cycle;
configuring a second MAC unit within the processor in pipelined fashion, wherein the second MAC unit accepts input elements of the first matrix and the second matrix, wherein the first element of the first matrix and a second element of the second matrix are presented to the second MAC unit on a second cycle, and wherein a second element of the first matrix and the first element of the second matrix are presented to the first MAC unit on the second cycle;
further configuring one or more additional MAC units within the processor in pipelined fashion; and
executing multiply-accumulate operations in pipelined fashion.
23. A computer system for data manipulation comprising:
a memory which stores instructions;
one or more processors coupled to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to:
obtain a first matrix with dimensions m×k and a second matrix with dimensions k×n for matrix multiplication within a processor;
configure a first matrix multiply-accumulate (MAC) unit within the processor, wherein the first MAC unit accepts input elements of the first matrix and the second matrix, and wherein a first element of the first matrix and a first element of the second matrix are presented to the first MAC unit on a first cycle;
configure a second MAC unit within the processor in pipelined fashion, wherein the second MAC unit accepts input elements of the first matrix and the second matrix, wherein the first element of the first matrix and a second element of the second matrix are presented to the second MAC unit on a second cycle, and wherein a second element of the first matrix and the first element of the second matrix are presented to the first MAC unit on the second cycle;
further configure one or more additional MAC units within the processor in pipelined fashion; and
execute multiply-accumulate operations in pipelined fashion.