Compiling tensor operators for neural network models based on tensor tile configurations
Embodiments of the present disclosure include systems and methods for compiling tensor operators for neural network models based on tensor tile configurations. A tensor expression for a neural network model is received. A tensor tile configuration for the tensor expression is determined based on specifications associated with a set of hardware devices. Based on the tensor tile configuration, a set of code for implementing the tensor expression for the neural network model is generated. The set of code is for execution on the set of hardware.
1 . A non-transitory machine-readable medium storing a program executable by at least one processing unit of a device, the program comprising sets of instructions for:
receiving a tensor expression for a neural network model, wherein the tensor expression specifies an operator that takes a set of input tensors, performs computations on the set of input tensors, and stores a result of the computations in an output tensor;
determining a tensor tile configuration for the tensor expression based on specifications associated with a set of hardware devices, the tensor tile configuration comprising a plurality of tensor tile shapes associated with the set of input tensors and the output tensor, the plurality of tensor tile shapes optimized for a plurality of memory layers of a memory architecture of a particular hardware device, wherein a different valid tensor tile shape with a highest memory metric is determined for each memory layer of the plurality of memory layers, wherein each tensor tile shape of the plurality of tensor tile shapes is associated with dimensions inferred from a tensor tile shape definition, the tensor tile shape definition associated with the tensor expression, and wherein the different valid tensor tile shape is determined based on a set of defined rules; and
based on the tensor tile configuration comprising the plurality of tensor tile shapes, generating a set of code for implementing the tensor expression for the neural network model, the set of code for execution on the plurality of memory layers of the particular hardware device of the set of hardware devices.
2 . The non-transitory machine-readable medium of claim 1 , wherein the specifications associated with the set of hardware devices include a number of threads that can be processed concurrently by the particular hardware device in the set of hardware devices, wherein determining the tensor tile configuration for the tensor expression comprises determining a size of a shape of a tensor tile in the tensor tile configuration that is a multiple of the number of threads that can be processed concurrently by the particular hardware device.
3 . The non-transitory machine-readable medium of claim 1 , wherein the specifications associated with the set of hardware devices include a length of a memory transaction utilized by a hardware device in the set of hardware devices, wherein determining the tensor tile configuration for the tensor expression comprises determining a size of an inner-most dimension of a tensor tile that is a multiple of the length of a memory transaction.
4 . The non-transitory machine-readable medium of claim 1 , wherein the specifications associated with the set of hardware devices include a number of memory banks into which memory included in a hardware device is divided and a width of memory banks, wherein the program further comprises instructions for determining an amount of padding for the tensor expression based on the number of memory banks and the width of memory banks.
5 . The non-transitory machine-readable medium of claim 1 , wherein the tensor expression specifying a set of shapes for the set of input tensors, a shape for the output tensor, and a set of tensor operations operating on the set of input tensors to generate the output tensor.
6 . The non-transitory machine-readable medium of claim 5 , wherein determining the tensor tile configuration for the tensor expression comprises determining a shape of a tensor tile into which the shape of an input tensor in the set of input tensors can be evenly partitioned.
7 . The non-transitory machine-readable medium of claim 1 , wherein determining the tensor tile configuration for the tensor expression comprises replicating the tensor tile configuration for each hardware device in the set of hardware devices other than the particular hardware device.
8 . A method comprising:
receiving a tensor expression for a neural network model, wherein the tensor expression specifies an operator that takes a set of input tensors, performs computations on the set of input tensors, and stores a result of the computations in an output tensor;
determining a tensor tile configuration for the tensor expression based on specifications associated with a set of hardware devices, the tensor tile configuration comprising a plurality of tensor tile shapes associated with the set of input tensors and the output tensor, the plurality of tensor tile shapes optimized for a plurality of memory layers of a memory architecture of a particular hardware device of the set of hardware devices, wherein a different valid tensor tile shape with a highest memory metric is determined for each memory layer of the plurality of memory layers, wherein each tensor tile shape of the plurality of tensor tile shapes is associated with dimensions inferred from a tensor tile shape definition, the tensor tile shape definition associated with the tensor expression, and wherein the different valid tensor tile shape is determined based on a set of defined rules; and
based on the tensor tile configuration comprising the plurality of tensor tile shapes, generating a set of code for implementing the tensor expression for the neural network model, the set of code for execution on the plurality of memory layers of the particular hardware device of the set of hardware devices.
9 . The method of claim 8 , wherein the specifications associated with the set of hardware devices include a number of threads that can be processed concurrently by the particular hardware device in the set of hardware devices, wherein determining the tensor tile configuration for the tensor expression comprises determining a size of a shape of a tensor tile in the tensor tile configuration that is a multiple of the number of threads that can be processed concurrently by the particular hardware device.
10 . The method of claim 8 , wherein the specifications associated with the set of hardware devices include a length of a memory transaction utilized by a hardware device in the set of hardware devices, wherein determining the tensor tile configuration for the tensor expression comprises determining a size of an inner-most dimension of a tensor tile that is a multiple of the length of a memory transaction.
11 . The method of claim 8 , wherein the specifications associated with the set of hardware devices include a number of memory banks into which memory included in a hardware device is divided and a width of memory banks, the method further comprising determining an amount of padding for the tensor expression based on the number of memory banks and the width of memory banks.
12 . The method of claim 8 , wherein the tensor expression specifying a set of shapes for the set of input tensors, a shape for the output tensor, and a set of tensor operations operating on the set of input tensors to generate the output tensor.
13 . The method of claim 12 , wherein determining the tensor tile configuration for the tensor expression comprises determining a shape of a tensor tile into which the shape of an input tensor in the set of input tensors can be evenly partitioned.
14 . The method of claim 8 , wherein determining the tensor tile configuration for the tensor expression comprises replicating the tensor tile configuration for each hardware device in the set of hardware devices other than the particular hardware device.
15 . A system comprising:
a set of processing units; and
a non-transitory machine-readable medium storing instructions that when executed by at least one processing unit in the set of processing units cause the at least one processing unit to:
receive a tensor expression for a neural network model, wherein the tensor expression specifies an operator that takes a set of input tensors, performs computations on the set of input tensors, and stores a result of the computations in an output tensor;
determine a tensor tile configuration for the tensor expression based on specifications associated with a set of hardware devices, the tensor tile configuration comprising a plurality of tensor tile shapes associated with the set of input tensors and the output tensor, the plurality of tensor tile shapes optimized for a plurality of memory layers of a memory architecture of a particular hardware device of the set of hardware devices, wherein a different valid tensor tile shape with a highest memory metric is determined for each memory layer of the plurality of memory layers, wherein each tensor tile shape of the plurality of tensor tile shapes is associated with dimensions inferred from a tensor tile shape definition, the tensor tile shape definition associated with the tensor expression, and wherein the different valid tensor tile shape is determined based on a set of defined rules; and
based on the tensor tile configuration comprising the plurality of tensor tile shapes, generate a set of code for implementing the tensor expression for the neural network model, the set of code for execution on the plurality of memory layers of the particular hardware device of the set of hardware devices.
16 . The system of claim 15 , wherein the specifications associated with the set of hardware devices include a number of threads that can be processed concurrently by the particular hardware device in the set of hardware devices, wherein determining the tensor tile configuration for the tensor expression comprises determining a size of a shape of a tensor tile in the tensor tile configuration that is a multiple of the number of threads that can be processed concurrently by the particular hardware device.
17 . The system of claim 15 , wherein the specifications associated with the set of hardware devices include a length of a memory transaction utilized by a hardware device in the set of hardware devices, wherein determining the tensor tile configuration for the tensor expression comprises determining a size of an inner-most dimension of a tensor tile that is a multiple of the length of a memory transaction.
18 . The system of claim 15 , wherein the specifications associated with the set of hardware devices include a number of memory banks into which memory included in a hardware device is divided and a width of memory banks, wherein the instructions further cause the at least one processing unit to determine an amount of padding for the tensor expression based on the number of memory banks and the width of memory banks.
19 . The system of claim 15 , wherein the tensor expression specifying a set of shapes for the set of input tensors, a shape for the output tensor, and a set of tensor operations operating on the set of input tensors to generate the output tensor.
20 . The system of claim 19 , wherein determining the tensor tile configuration for the tensor expression comprises determining a shape of a tensor tile into which the shape of an input tensor in the set of input tensors can be evenly partitioned.