Dynamic processing element array expansion
A computer-implemented method includes receiving a neural network model that includes a tensor operation, dividing the tensor operation into a set of sub-operations, and generating instructions for performing a plurality of sub-operations of the set of sub-operations on respective computing engines of a plurality of computing engines on a same integrated circuit device or on different integrated circuit devices. Each sub-operation of the set of sub-operations generates a portion of a final output of the tensor operation. An inference is made based on a result of a sub-operation of the plurality of sub-operations, or based on results of the plurality of sub-operations.
1. A computer-implemented method, comprising:
receiving a neural network model that includes a first tensor operation between a first tensor and a second tensor;
dividing the first tensor operation into a set of sub-operations, wherein each sub-operation of the set of sub-operations generates a portion of a final output of the first tensor operation, and wherein dividing the first tensor operation comprises one of:
assigning each sub-operation to a respective portion of the first tensor,
assigning each sub-operation to a respective portion of the second tensor, or
assigning each sub-operation to both a respective portion of the first tensor and a respective portion of the second tensor; and
generating instructions for performing individual sub-operations of the set of sub-operations on respective computing engines of a plurality of computing engines.
2. The computer-implemented method of claim 1 , wherein the instructions include instructions for performing the set of sub-operations in parallel on the plurality of computing engines.
3. The computer-implemented method of claim 1 , wherein the first tensor operation includes a multi-layer perceptron operation.
4. The computer-implemented method of claim 1 , wherein the first tensor operation generates, using a set of input feature maps contained in the first tensor and a set of filters contained in the second tensor, an output tensor including a set of output feature maps.
5. The computer-implemented method of claim 4 , wherein:
the set of input feature maps includes C input feature maps for each input dataset of N input datasets;
the set of filters includes M filters each including C filter matrices; and
the set of output feature maps includes M output feature maps for each input dataset of the N input datasets, wherein each output feature map of the M output feature maps for the input dataset is a result of a convolution between the C input feature maps of the input dataset and the C filter matrices of a respective filter of the M filters.
6. The computer-implemented method of claim 5 , wherein each computing engine of the plurality of computing engines includes a two-dimensional array of processing elements characterized by:
a total number of columns of the two-dimensional array of processing elements being less than M; or
a total number of rows of the two-dimensional array of processing elements being less than C.
7. The computer-implemented method of claim 4 , wherein each sub-operation of the set of sub-operations generates a portion of the set of output feature maps.
8. The computer-implemented method of claim 7 , wherein the portion of the set of output feature maps includes a subset of output feature maps in the set of output feature maps.
9. The computer-implemented method of claim 8 , wherein each sub-operation of the set of sub-operations generates the portion of the set of output feature maps using:
the entire set of input feature maps as represented by the first tensor; and
a subset of filters of the set of filters, as represented by a respective portion of the second tensor.
10. The computer-implemented method of claim 7 , wherein the portion of the set of output feature maps includes a portion of each output feature map of a subset of output feature maps in the set of output feature maps.
11. The computer-implemented method of claim 10 , wherein each sub-operation of the set of sub-operations generates the portion of the set of output feature maps using:
the entire set of filters as represented by the second tensor; and
a portion of each input feature map of a subset of input feature maps in the set of input feature maps, as represented by a respective portion of the first tensor.
12. The computer-implemented method of claim 10 , wherein each sub-operation of the set of sub-operations generates the portion of the set of output feature maps using:
a subset of filters in the set of filters, as represented by a respective portion of the second tensor; and
a portion of each input feature map of a subset of input feature maps in the set of input feature maps, as represented by a respective portion of the first tensor.
13. The computer-implemented method of claim 1 , further comprising generating instructions for making an inference based on a result of at least one sub-operation of the set of sub-operations.
14. The computer-implemented method of claim 1 , wherein the instructions configure the plurality of computing engines to perform the set of sub-operations in parallel.
15. The computer-implemented method of claim 1 , further comprising generating instructions for saving an output of each sub-operation of the set of sub-operations into a respective sub-region of a set of sub-regions of a region of a memory device.
16. The computer-implemented method of claim 1 , further comprising:
identifying a second tensor operation in the neural network model, the second tensor operation and the first tensor operation having no data dependency;
dividing the second tensor operation into a second set of sub-operations, wherein each sub-operation of the second set of sub-operations generates a portion of a final output of the second tensor operation;
generating instructions for performing, in parallel with performing the set of sub-operations of the first tensor operation, individual sub-operations of the second set of sub-operations on respective computing engines of a second plurality of computing engines; and
generating instructions for making an inference based on a result of the first tensor operation and a result of the second tensor operation.
17. The computer-implemented method of claim 1 , wherein the final output of the first tensor operation is an output tensor generated through concatenating results of the individual sub-operations.
18. A non-transitory computer-readable medium having instructions stored therein, wherein the instructions, when executed by one or more processors, cause the one or more processors to perform operations including:
receiving a neural network model that includes a first tensor operation between a first tensor and a second tensor;
dividing the first tensor operation into a set of sub-operations, wherein each sub-operation of the set of sub-operations generates a portion of a final output of the first tensor operation, and wherein dividing the first tensor operation comprises one of:
assigning each sub-operation to a respective portion of the first tensor,
assigning each sub-operation to a respective portion of the second tensor, or
assigning each sub-operation to both a respective portion of the first tensor and a respective portion of the second tensor; and
generating instructions for performing individual sub-operations of the set of sub-operations on respective computing engines of a plurality of computing engines.
19. The non-transitory computer-readable medium of claim 18 , wherein the instructions further cause the one or more processors to perform operations including:
generating instructions for making an inference based on a result of at least one sub-operation of the set of sub-operations,
wherein the plurality of computing engines includes computing engines in a same integrated circuit device or in different integrated circuit devices.
20. The non-transitory computer-readable medium of claim 18 , wherein the instructions further cause the one or more processors to perform operations including:
identifying a second tensor operation in the neural network model, the second tensor operation and the first tensor operation having no data dependency;
dividing the second tensor operation into a second set of sub-operations, wherein each sub-operation of the second set of sub-operations generates a portion of a final output of the second tensor operation;
generating instructions for performing, in parallel with performing the set of sub-operations of the first tensor operation individual sub-operations of the second set of sub-operations on respective computing engines of a second plurality of computing engines; and
generating instructions for making an inference based on a result of the first tensor operation and a result of the second tensor operation.