Forward-style gradient GeMMs
A method for improving runtime performance and alleviating place and route issues in a reconfigurable computing system includes receiving a compute graph for execution on a reconfigurable dataflow processor. The compute graph includes a node specifying a template-based operation on a first and second tensor having a shared batch dimension B. The node may be split into B nodes. Each of the template-based operations on the pair of tensors may be replace with a GeMM operation on the first reduced rank tensor slice and a tile. B nodes that specify the GeMM operation may be appended with at least one first addition node that accepts input from the B nodes to produce a first modified compute graph. The first modified compute graph may be executed. The method describes a significant improvement to overall compute utilization across gradient-sections. Spatial tiling of tensors facilitates gradient calculation without the use of accumulators.
1 . A system for reducing latency and increasing throughput in reconfigurable dataflow processors, the system comprising:
a host computer comprising a graph optimization module configured to conduct a method comprising:
receiving a compute graph for execution on a reconfigurable dataflow processor (RDP), the compute graph comprising a node specifying a template-based gradient operation on a first tensor and a second tensor, the first and second tensors having a shared batch dimension of length B;
splitting the node into B nodes that each specify the template-based operation the first and second tensors;
replacing each of the template-based operations on a pair of tensors with a general matrix multiplication (GeMM) operation on a first reduced rank tensor slice and a tile to produce B nodes that specify the GeMM operation;
appending the B nodes that specify the GeMM operation with at least one first addition (ADDN) node that accepts input from the B nodes that specify the GeMM operation to produce a first modified compute graph; and
a reconfigurable dataflow processor (RDP) configured to execute the first modified compute graph.
2 . The system of claim 1 , further comprising slicing the first and second tensors on the shared batch dimension of length B to produce B first tensor slices and B second tensor slices, each of the first and second tensors of the B first tensor slices and B second tensor slices having a batch dimension of length 1.
3 . The system of claim 2 , further comprising reshaping each of the first and second tensors of the B first tensor slices and B second tensor slices to produce B reduced rank first tensor slices and B reduced rank second tensor slices, a reduced rank first tensor slice of the B reduced rank first tensor slices corresponding to the reduced rank first tensor slice in the GeMM operations.
4 . The system of claim 3 , further comprising tiling each reduced rank second tensor slice of the B reduced rank second tensor slices to produce B tile sets, each tile set of the B tile sets having indexed tile elements.
5 . The system of claim 4 , wherein each tile of the B tile sets has an equal block size.
6 . The system of claim 4 , wherein the GeMM operation is performed on a pair of tiles in the B tile sets, wherein the pair of tiles have index-paired tile elements.
7 . The system of claim 1 , wherein one or more of the B nodes that specify the GeMM operation on the first reduced rank tensor and the tile comprise a section.
8 . The system of claim 7 , wherein each output sub-tensor from each node of the B nodes of a first section are locally summed by at least one first ADDN node to generate a first partial sum, each output sub-tensor from each node of the B nodes of a second section are locally summed (by at least one ADDN node) to generate a second partial sum, and the first partial sum and the second partial sum are locally summed by a second addition (ADDN) node to generate a first intermediate sum.
9 . The system of claim 8 , wherein the first partial sum is read directly from Dynamic Random-Access Memory (DRAM) and the second partial sum is read from a pattern memory unit (PMU).
10 . A method for reducing latency and increasing throughput in a reconfigurable computing system, the method comprising:
receiving a compute graph for execution on a reconfigurable dataflow processor (RDP), the compute graph comprising a node specifying a template-based gradient operation on a first tensor and a second tensor, the first and second tensors having a shared batch dimension of length B;
splitting the node into B nodes that each specify the template-based operation the first and second tensors;
replacing each of the template-based operations on a pair of tensors with a general matrix multiplication (GeMM) operation on a first reduced rank tensor slice and a tile to produce B nodes that specify the GeMM operation;
appending the B nodes that specify the GeMM operation with at least one first addition (ADDN) node that accepts input from the B nodes that specify the GeMM operation to produce a first modified compute graph; and
executing the first modified compute graph on the RDP.
11 . The method of claim 10 , further comprising slicing the first and second tensors on the shared batch dimension of length B to produce B first tensor slices and B second tensor slices, each of the first and second tensors of the B first tensor slices and B second tensor slices having a batch dimension of length 1.
12 . The method of claim 11 , further comprising reshaping each of the first and second tensors of the B first tensor slices and B second tensor slices to produce B reduced rank first tensor slices and B reduced rank second tensor slices, a reduced rank first tensor slice of the B reduced rank first tensor slices corresponding to the reduced rank first tensor slice in the GeMM operations.
13 . The method of claim 12 , further comprising tiling each reduced rank second tensor slice of the B reduced rank second tensor slices to produce B tile sets, each tile set of the B tile sets having indexed tile elements.
14 . The method of claim 13 , wherein each tile of the B tile sets has an equal block size.
15 . The method of claim 13 , wherein the GeMM operation is performed on a pair of tiles in the B tile sets, wherein the pair of tiles have index-paired tile elements.
16 . The method of claim 10 , wherein one or more of the B nodes that specify the GeMM operation on the first reduced rank tensor and the tile comprise a section.
17 . The method of claim 16 , wherein each output sub-tensor from each node of the B nodes of a first section are locally summed by at least one first ADDN node to generate a first partial sum, each output sub-tensor from each node of the B nodes of a second section are locally summed by at least one first ADDN node to generate a second partial sum, and the first partial sum and the second partial sum are locally summed by a second addition (ADDN) node to generate a first intermediate sum.
18 . The method of claim 17 , wherein the first partial sum is read directly from Dynamic Random-Access Memory (DRAM) and the second partial sum is read from a pattern memory unit (PMU).
19 . A non-transitory computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, wherein the non-transitory computer readable storage medium is not a transitory signal per se, wherein the program instructions are executable by a processor to cause the processor to conduct a method comprising:
receiving a compute graph for execution on a reconfigurable dataflow processor (RDP), the compute graph comprising a node specifying a template-based gradient operation on a first tensor and a second tensor, the first and second tensors having a shared batch dimension of length B;
splitting the node into B nodes that each specify the template-based operation the first and second tensors;
replacing each of the template-based operations on a pair of tensors with a general matrix multiplication (GeMM) operation on a first reduced rank tensor slice and a tile to produce B nodes that specify the GeMM operation;
appending the B nodes that specify the GeMM operation with at least one first addition (ADDN) node that accepts input from the B nodes that specify the GeMM operation to produce a first modified compute graph; and
executing the first modified compute graph on the RDP.