Optimizing tensor tiling in neural networks based on a tiling cost model
A method comprises a compiler analyzing a graph to determine a pipeline of operators based on a shared dimension of input and output tensors among the operators. The operators are included in the graph and the graph corresponds to a dataflow application. The compiler determines a tiling decision associated with the pipeline and a tiling cost associated with the tiling decision. The tiling decision can comprise a tile shape to slice tensors of operators of the pipeline. Based on the tiling cost, the compiler determines that the tiling decision improves an optimization objective and includes the pipeline and tiling decision in mapping decisions associated with executing the application on a computing system. The compiler can apply a tiling cost model to determine the tiling costs. A computer program product and a computing system can implement the method.
1 . A method, the method comprising:
determining, by a compiler of a first computing system, based on a first shared dimension of output and input tensors of a first set of operators, a first pipeline comprising the first set of operators, the first set of operators among operators included in a graph, the operators included in the graph comprising operators of a dataflow application, wherein a given pair of consecutive operators in the first set of operators is configured such that an output tensor of the first operator in the given pair of consecutive operators has a first dimension equal to the first shared dimension, an input tensor to the second operator in the given pair of consecutive operators has a second dimension also equal to the first shared dimension, and wherein by sharing the first shared dimension, the first set of operators performs computations in parallel;
determining, by the compiler, a first tiling decision associated with the first pipeline, wherein the first tiling decision is based at least on the first shared dimension, and wherein the first tiling decision specifies a tile size and an overlap amount for slicing tensors shared between consecutive operators in the first pipeline;
determining, by the compiler, a first tiling cost associated with the first tiling decision, the first tiling cost corresponding to a first optimization objective, wherein determining the first tiling cost comprises computing, by the compiler using a tiling cost model, a metric indicative of at least one of memory utilization efficiency, processor utilization, and computational parallelism achievable when executing the first pipeline on the second computing system using the tile size and overlap amount;
determining, by the compiler, based on the first tiling cost, that the first tiling decision improves the first optimization objective; and
including, by the compiler, based on the determining that the first tiling decision improves the first optimization objective, the first pipeline and the first tiling decision among mapping decisions associated with executing the dataflow application by a second computing system, wherein the first tiling decision increases parallelization of computations and processor utilization on the second computing system by enabling the second computing system to process multiple tiles of the tensors concurrently across processing elements of the second computing system.
2 . The method of claim 1 , wherein the first pipeline comprises a nested pipeline.
3 . The method of claim 1 , wherein the method further comprises:
determining, by the compiler, a second tiling decision associated with an operator among the operators included in the graph;
determining, by the compiler, a second tiling cost associated with the operator, the second tiling cost corresponding to a second optimization objective;
determining, by the compiler, based on the second tiling cost, that the second tiling decision improves the second optimization objective; and
including, by the compiler, based on the determining that the second tiling decision improves the second optimization objective, the operator and the second tiling decision among mapping decisions associated with executing the dataflow application by the second computing system.
4 . The method of claim 1 , the method further comprising:
determining, by the compiler, based on a second shared dimension of output and input tensors of a second set of operators among the operators included in the graph, a second pipeline comprising the second set of operators;
determining, by the compiler, a second tiling decision associated with the second pipeline;
determining, by the compiler, a second tiling cost corresponding to the second tiling decision, the second tiling cost based on a second optimization objective;
determining, by the compiler, based on the second tiling cost, that the second tiling decision does not improve a second optimization; and
excluding, by the compiler, based on the determining that the second tiling decision does not improve the second optimization objective, the second pipeline from among the mapping decisions associated with executing the dataflow application by the second computing system.
5 . The method of claim 1 , wherein the first tiling decision comprises a first tile shape to slice an output tensor of a first operator, included in the first pipeline, the output tensor comprising an input tensor to a second operator included in the first pipeline.
6 . The method of claim 1 , wherein the method of the compiler determining the first tiling cost comprises determining, by the compiler, the first tiling cost using a tiling cost model to compute the first tiling cost.
7 . The method of claim 1 , wherein the method of determining, by the compiler, based on the first tiling cost, that the first tiling decision improves the first optimization objective comprises comparing, by the compiler, the first tiling cost to a threshold value of an optimization metric associated with the first optimization objective.
8 . The method of claim 1 , wherein the first optimization objective comprises a memory optimization objective selected from a group consisting of: a first tile shape fitting in a first memory of the second computing system; increasing a utilization of a second memory of the second computing system; reducing a number of stage buffers among a first producer operator and a first consumer operator included in the first pipeline; and, reducing a size of a stage buffer among a second producer operator and a second consumer operator.
9 . The method of claim 1 , wherein the first optimization objective comprises a processing optimization objective selected from a group consisting of: increasing a number of operators comprising the first pipeline; increasing a number of parallel operations performed by the second computing system to execute the dataflow application; increasing a utilization of a first processor of the second computing system to execute the dataflow application; and, balancing pipeline stages in the first pipeline.
10 . A computer program product, the computer program product comprising a computer readable storage medium having first program instructions embodied therewith, wherein the first program instructions are executable by at least one processor to cause the at least one processor to:
determine, based on a first shared dimension of output and input tensors of a first set of operators, a first pipeline comprising the first set of operators, the first set of operators among operators included in a graph, the operators included in the graph comprising operators of a dataflow application, wherein a given pair of consecutive operators in the first set of operators is configured such that an output tensor of the first operator in the given pair of consecutive operators has a first dimension equal to the first shared dimension, and an input tensor to the second operator in the given pair of consecutive operators has a second dimension also equal to the first shared dimension, and wherein by sharing the first shared dimension, the first set of operators performs computations in parallel;
determine a first tiling decision associated with the first pipeline, wherein the first tiling decision is based at least on the first shared dimension, and wherein the first tiling decision specifies a tile size and an overlap amount for slicing tensors shared between consecutive operators in the first pipeline;
determine, a first tiling cost associated with the first tiling decision, the first tiling cost corresponding to a first optimization objective, wherein determining the first tiling cost comprises computing, using a tiling cost model, a metric indicative of at least one of memory utilization efficiency, processor utilization, and computational parallelism achievable when executing the first pipeline on the second computing system using the tile size and overlap amount;
determine, based on the first tiling cost, that the first tiling decision improves the first optimization objective; and
include, based on the determining that the first tiling decision improves the first optimization objective, the first pipeline and the first tiling decision among mapping decisions associated with executing the dataflow application by a second computing system, wherein the first tiling decision increases parallelization of computations and processor utilization on the second computing system by enabling the second computing system to process multiple tiles of the tensors concurrently across processing elements of the second computing system.
11 . The computer program product of claim 10 , wherein the first program instructions are executable by the at least one processor to further cause the at least one processor to:
determine the first tiling cost using a tiling cost model to compute the first tiling cost.
12 . The computer program product of claim 10 , wherein the first program instructions are executable by the at least one processor to further cause the at least one processor to:
determine a second tiling decision associated with an operator among the operators included in the graph;
determine a second tiling cost associated with the operator, the second tiling cost corresponding to a second optimization objective;
determine, based on the second tiling cost, that the second tiling decision does not improve a second optimization objective; and
exclude, based on the determining that the second tiling decision improves the second optimization objective, the operator and the second tiling decision among mapping decisions associated with executing the dataflow application by the second computing system.
13 . A first computing system, the first computing system comprising:
a processor and a compiler, the compiler configured to execute on the processor to:
determine, based on a first shared dimension of output and input tensors of a first set of operators, a first pipeline comprising the first set of operators, the first set of operators among operators included in a graph, the operators included in the graph comprising operators of a dataflow application, wherein a given pair of consecutive operators in the first set of operators is configured such that an output tensor of the first operator in the given pair of consecutive operators has a first dimension equal to the first shared dimension, and an input tensor to the second operator in the given pair of consecutive operators has a second dimension also equal to the first shared dimension, and wherein by sharing the first shared dimension, the first set of operators performs computations in parallel;
determine a first tiling decision associated with the first pipeline, wherein the first tiling decision is based at least on the first shared dimension, and wherein the first tiling decision specifies a tile size and an overlap amount for slicing tensors shared between consecutive operators in the first pipeline;
determine a first tiling cost associated with the first tiling decision, the first tiling cost corresponding to a first optimization objective, wherein determining the first tiling cost comprises computing, using a tiling cost model, a metric indicative of at least one of memory utilization efficiency, processor utilization, and computational parallelism achievable when executing the first pipeline on the second computing system using the tile size and overlap amount;
determine, based on the first tiling cost, that the first tiling decision improves the first optimization objective; and
include, based on the determining that the first tiling decision improves the first optimization objective, the first pipeline and the first tiling decision among mapping decisions associated with executing the dataflow application by a second computing system, wherein the first tiling decision increases parallelization of computations and processor utilization on the second computing system by enabling the second computing system to process multiple tiles of the tensors concurrently across processing elements of the second computing system.
14 . The first computing system of claim 13 , wherein the first pipeline comprises a nested pipeline.
15 . The first computing system of claim 13 , wherein the compiler is further configured to execute on the processor to:
determine a second tiling decision associated with an operator among the operators included in the graph;
determine a second tiling cost associated with the operator, the second tiling cost corresponding to a second optimization objective;
determine, based on the second tiling cost, that the second tiling decision improves the second optimization objective; and
include, based on the determining that the second tiling decision improves the second optimization objective, the operator and the second tiling decision among mapping decisions associated with executing the dataflow application by the second computing system.
16 . The first computing system of claim 13 , wherein the compiler is further configured to execute on the processor to:
determine, based on a second shared dimension of output and input tensors of a second set of operators among the operators included in the graph, a second pipeline comprising the second set of operators;
determine a second tiling decision associated with the second pipeline;
determine a second tiling cost corresponding to the second tiling decision, the second tiling cost based on a second optimization objective;
determine, based on the second tiling cost, that the second tiling decision does not improve a second optimization; and
exclude, based on the determining that the second tiling decision does not improve the second optimization objective, the second pipeline from among the mapping decisions associated with executing the dataflow application by the second computing system.
17 . The first computing system of claim 13 , wherein the first tiling decision comprises a first tile shape to slice an output tensor of a first operator, included in the first pipeline, the output tensor comprising an input tensor to a second operator included in the first pipeline.
18 . The first computing system of claim 13 , wherein the compiler configured to execute on the processor to determine the first tiling cost comprises the compiler further configured to execute on the processor to determine the first tiling cost by comparing the first tiling cost to a threshold value of an optimization metric associated with the first optimization objective.
19 . The first computing system of claim 13 , wherein the first optimization objective comprises a memory optimization objective selected from a group consisting of: a first tile shape fitting in a first memory of the second computing system; increasing a utilization of a second memory of the second computing system; reducing a number of stage buffers among a first producer operator and a first consumer operator included in the first pipeline; and, reducing a size of a stage buffer among a second producer operator and a second consumer operator.
20 . The first computing system of claim 13 , wherein the first optimization objective comprises a processing optimization objective selected from a group consisting of: increasing a number of operators comprising the first pipeline; increasing a number of parallel operations performed by the second computing system to execute the dataflow application; increasing a utilization of a first processor of the second computing system to execute the dataflow application; and, balancing pipeline stages in the first pipeline.