Compiler managed tensor parallel execution
Techniques for implementing tensor parallel execution can include identifying a first tensor contraction operation in a compute flow, and slicing the first tensor contraction operation into a first set of multiple tensor contraction portions to have each compute engine of multiple compute engines perform a portion of the first tensor contraction operation. A set of slicing options can then be determined for a second tensor contraction operation that operates on a tensor result of the first tensor contraction operation. A cost for each slicing option is determined, and a slicing option having the lowest cost is selected. The second tensor contraction operation is sliced according to the selected slicing option to have each compute engine perform a portion of the second tensor contraction operation. Collective compute operations can be inserted in the compute flow for the first and second tensor contraction operations.
1. A computer-implemented method for compiling a neural network model for execution on a neural network acceleration engine having multiple accelerators, the method comprising:
obtaining a description of the neural network model;
generating a representation of a compute flow graph of the neural network model, the compute flow graph having nodes corresponding to tensor operations and edges corresponding to data dependencies;
performing data layout assignment for tensors used in the tensor operations;
performing a compute-based slicing of the compute flow graph by:
identifying a set of tensor contraction operations in the compute flow graph;
for each of the tensor contraction operations:
selecting a slicing option having a lowest cost to slice the tensor contraction operation into multiple tensor contraction portions, wherein a cost for slicing a tensor contraction operation includes an overhead to distribute tensor data to the multiple accelerators;
slicing the tensor contraction operation according to the selected slicing option;
propagating the selected slicing option to slice other tensor operations; and
assigning the multiple tensor contraction portions and sliced tensor operations to the multiple accelerators, respectively, to have each accelerator perform a portion of the tensor contraction operation and sliced tensor operations; and
inserting collective compute operations into the representation of the neural network model to facilitate performing the tensor contraction operations in the multiple accelerators; and
generating machine instructions for execution of the neural network model on the multiple accelerators based on the compute-based slicing,
wherein the machine instructions are executed on the multiple accelerators to implement the neural network model.
2. The computer-implemented method of claim 1 , wherein the cost for slicing a tensor contraction operation that operates on a tensor result from a prior tensor contraction operation includes an overhead to resolve slicing mismatch between the tensor contraction operation and the prior tensor contraction operation.
3. The computer-implemented method of claim 1 , further including performing additional tensor slicing after the compute-based slicing to keep memory usage of each accelerator to be within a memory capacity of the corresponding accelerator.
4. A computer-implemented method comprising:
identifying a first tensor contraction operation in a compute flow;
slicing the first tensor contraction operation into a first set of multiple tensor contraction portions to have each compute engine of a plurality of compute engines perform a portion of the first tensor contraction operation;
determining a set of slicing options for a second tensor contraction operation that operates on a tensor result of the first tensor contraction operation;
determining a cost for each slicing option in the set of slicing options for the second tensor contraction operation, wherein the cost for slicing the second tensor contraction operation includes an overhead to distribute tensor data to the plurality of compute engines;
selecting a slicing option having a lowest cost;
slicing the second tensor contraction operation according to the selected slicing option to have each compute engine perform a portion of the second tensor contraction operation;
inserting collective compute operations in the compute flow for the first and second tensor contraction operations; and
generating machine instructions for the compute flow, wherein the machine instructions are executed on the plurality of compute engines to perform the compute flow.
5. The computer-implemented method of claim 4 , wherein the first tensor contraction operation is sliced along a left tensor free axis, and the collective compute operations include scattering a left tensor of the first tensor contraction operation, and duplicating a right tensor of the first tensor contraction operation.
6. The computer-implemented method of claim 4 , wherein the first tensor contraction operation is sliced along a right tensor free axis, and the collective compute operations include duplicating a left tensor of the first tensor contraction operation, and scattering a right tensor of the first tensor contraction operation.
7. The computer-implemented method of claim 4 , wherein the first tensor contraction operation is sliced along a contracted axis, and the collective compute operations include scattering a left tensor of the first tensor contraction operation, and scattering a right tensor of the first tensor contraction operation.
8. The computer-implemented method of claim 4 , wherein determining the cost for each slicing option of the second tensor contraction operation includes determining a cost to resolve slicing mismatch on the tensor result of the first tensor contraction operation.
9. The computer-implemented method of claim 8 , wherein no additional collective compute operation is used to resolve slicing mismatch when slicing both the first and second tensor contraction operations along respective left tensor free axes, or slicing the first tensor contraction operation along a right tensor free axis and the second tensor contraction operation along a contracted axis.
10. The computer-implemented method of claim 8 , wherein an all-to-all collective compute operation is added to resolve slicing mismatch when slicing the first tensor contraction operation along a left tensor free axis and the second tensor contraction operation along a contracted axis, or slicing the first tensor contraction operation along a right tensor free axis and the second tensor contraction operation along a left tensor free axis.
11. The computer-implemented method of claim 8 , wherein an all-gather collective compute operation is added to resolve slicing mismatch when slicing the first tensor contraction operation along a left tensor free axis and the second tensor contraction operation along a right tensor free axis, or slicing both the first and second tensor contraction operations along respective right tensor free axes.
12. The computer-implemented method of claim 8 , wherein an all-reduce collective compute operation is added to resolve slicing mismatch when slicing the first tensor contraction operation along a contracted axis and the second tensor contraction operation along a right tensor free axis.
13. The computer-implemented method of claim 8 , wherein a reduce-scatter collective compute operation is added to resolve slicing mismatch when slicing the first tensor contraction operation along a contracted axis and the second tensor contraction operation along a left tensor free axis, or slicing both the first and second tensor contraction operation along respective contracted axes.
14. The computer-implemented method of claim 4 , wherein the first and second tensor contraction operations are represented as affine loopnests.
15. The computer-implemented method of claim 14 , wherein slicing each of the first and second tensor contraction operations includes setting an axis being sliced to be an outermost loop of the respective affine loopnests, and parallelizing the outermost loop into multiple threads for the plurality of compute engines.
16. A non-transitory computer readable medium having stored therein instructions that, when executed by one or more processors, cause the one or more processors to execute a compiler, the compiler performing operations including: identifying a first tensor contraction operation in a compute flow;
slicing the first tensor contraction operation into a first set of multiple tensor contraction portions to have each compute engine of a plurality of compute engines perform a portion of the first tensor contraction operation;
determining a set of slicing options for a second tensor contraction operation that operates on a tensor result of the first tensor contraction operation;
determining a cost for each slicing option in the set of slicing options for the second tensor contraction operation, wherein the cost for slicing the second tensor contraction operation includes an overhead to distribute tensor data to the plurality of compute engines;
selecting a slicing option having a lowest cost;
slicing the second tensor contraction operation according to the selected slicing option to have each compute engine perform a portion of the second tensor contraction operation;
inserting collective compute operations in the compute flow for the first and second tensor contraction operations; and
generating machine instructions for the compute flow, wherein the machine instructions are executed on the plurality of compute engines to perform the compute flow.
17. The non-transitory computer readable medium of claim 16 , wherein determining the cost for each slicing option of the second tensor contraction operation includes determining a cost to resolve slicing mismatch on the tensor result of the first tensor contraction operation.
18. The non-transitory computer readable medium of claim 16 , wherein the first and second tensor contraction operations are represented as affine loopnests.
19. The non-transitory computer readable medium of claim 18 , wherein slicing each of the first and second tensor contraction operations includes setting an axis being sliced to be an outermost loop of the respective affine loopnests, and parallelizing the outermost loop into multiple threads for the plurality of compute engines.