Dynamic processing element array expansion
A computer-implemented method includes receiving a neural network model that includes a tensor operation, and dividing the tensor operation into sub-operations. The sub-operations includes at least two sub-operations that have no data dependency between the two sub-operations. The computer-implemented method further includes assigning a first sub-operation in the two sub-operations to a first computing engine, assigning a second sub-operation in the two sub-operations to a second computing engine, and generating instructions for performing, in parallel, the first sub-operation by the first computing engine and the second sub-operation by the second computing engine. An inference is then made based on a result of the first sub-operation, a result of the second sub-operation, or both. The first computing engine and the second computing engine are in a same integrated circuit device or in two different integrated circuit devices.
1. A computer-implemented method for implementing a neural network using two or more computing engines, the method comprising:
receiving a neural network model that includes a convolution operation for generating a set of output feature maps using a set of input feature maps and a set of filters;
dividing the convolution operation into two or more sub-operations that have no data dependency between each other, wherein each of the two or more sub-operations generates a portion of the set of output feature maps;
assigning a first sub-operation in the two or more sub-operations to a first computing engine in the two or more computing engines, wherein the first sub-operation generates a first portion of a final output of the convolution operation;
assigning a second sub-operation in the two or more sub-operations to a second computing engine in the two or more computing engines, wherein the second sub-operation generates a second portion of the final output of the convolution operation;
generating instructions for performing, in parallel, the first sub-operation by the first computing engine and the second sub-operation by the second computing engine; and
generating instructions for making an inference based on a result of the first sub-operation, a result of the second sub-operation, or both.
2. The computer-implemented method of claim 1 , wherein:
the set of input feature maps includes C input feature maps for each of N input datasets;
the set of filters includes M filters each including C filter matrices; and
the set of output feature maps includes M output feature maps for each of the N input datasets, wherein each of the M output feature maps for an input dataset is a result of a convolution between the C input feature maps of the input dataset and the C filter matrices in a filter of the M filters.
3. The computer-implemented method of claim 1 , wherein the portion of the set of output feature maps includes:
a first fraction of a total number of output feature maps in the set of output feature maps;
a first portion of each output feature map in a subset of the set of output feature maps; or
a second portion of each output feature map in a second fraction of the total number of output feature maps in the set of output feature maps.
4. The computer-implemented method of claim 3 , wherein the first sub-operation generates the portion of the set of output feature maps using:
the set of input feature maps and a first fraction of a total number of filters in the set of filters;
the set of filters and a first portion of each input feature map in a first subset of the set of input feature maps; or
a second fraction of the total number of filters in the set of filters and a second portion of each input feature map in a second subset of the set of input feature maps.
5. A computer-implemented method, comprising:
receiving a neural network model that includes a first tensor operation;
dividing the first tensor operation into sub-operations of the first tensor operation, the sub-operations including two sub-operations that have no data dependency between the two sub-operations;
assigning a first sub-operation in the two sub-operations to a first computing engine, wherein the first sub-operation generates a first portion of a final output of the first tensor operation;
assigning a second sub-operation in the two sub-operations to a second computing engine, wherein the second sub-operation generates a second portion of the final output of the first tensor operation; and
generating instructions for performing, in parallel, the first sub-operation by the first computing engine and the second sub-operation by the second computing engine.
6. The computer-implemented method of claim 5 , wherein the first tensor operation includes a multi-layer perceptron operation.
7. The computer-implemented method of claim 5 , wherein the first tensor operation generates, using a set of input feature maps and a set of filters, an output tensor including a set of output feature maps.
8. The computer-implemented method of claim 7 , wherein:
the set of input feature maps includes C input feature maps for each of N input datasets;
the set of filters includes M filters each including C filter matrices; and
the set of output feature maps includes M output feature maps for each of the N input datasets, wherein each of the M output feature maps for an input dataset is a result of a convolution between the C input feature maps of the input dataset and the C filter matrices in a filter of the M filters.
9. The computer-implemented method of claim 8 , wherein the first computing engine includes a two-dimensional processing element array characterized by:
a total number of columns of the two-dimensional processing element array less than M; or
a total number of rows of the two-dimensional processing element array less than C.
10. The computer-implemented method of claim 7 , wherein the first sub-operation generates a portion of the set of output feature maps.
11. The computer-implemented method of claim 10 , wherein the portion of the set of output feature maps includes a fraction of a total number of output feature maps in the set of output feature maps.
12. The computer-implemented method of claim 11 , wherein the first sub-operation generates the portion of the set of output feature maps using:
the set of input feature maps; and
a fraction of a total number of filters in the set of filters.
13. The computer-implemented method of claim 10 , wherein the portion of the set of output feature maps includes a portion of each output feature map in a subset of the set of output feature maps.
14. The computer-implemented method of claim 13 , wherein the first sub-operation generates the portion of the set of output feature maps using:
the set of filters; and
a portion of each input feature map in a subset of the set of input feature maps.
15. The computer-implemented method of claim 10 , wherein the portion of the set of output feature maps includes a portion of each output feature map in a fraction of a total number of output feature maps in the set of output feature maps.
16. The computer-implemented method of claim 15 , wherein the first sub-operation generates the portion of the set of output feature maps using:
a fraction of a total number of filters in the set of filters; and
a portion of each input feature map in a subset of the set of input feature maps.
17. The computer-implemented method of claim 5 , further comprising:
generating instructions for making an inference based on a result of the first sub-operation, a result of the second sub-operation, or both.
18. The computer-implemented method of claim 5 , further comprising:
identifying a second tensor operation in the neural network model, the second tensor operation and the first tensor operation having no data dependency;
dividing the second tensor operation into sub-operations of the second tensor operation, the sub-operations of the second tensor operation having no data dependency;
assigning a first sub-operation in the sub-operations of the second tensor operation to a third computing engine;
assigning a second sub-operation in the sub-operations of the second tensor operation to a fourth computing engine; and
generating instructions for performing, in parallel with the first sub-operation for the first tensor operation and the second sub-operation for the first tensor operation, the first sub-operation in the sub-operations of the second tensor operation by the third computing engine and the second sub-operation in the sub-operations of the second tensor operation by the fourth computing engine.
19. A non-transitory computer-readable medium having instructions stored therein, wherein the instructions, when executed by one or more processors, cause the one or more processors to perform operations including:
receiving a neural network model that includes a tensor operation;
dividing the tensor operation into sub-operations, the sub-operations including two sub-operations that have no data dependency between the two sub-operations;
assigning a first sub-operation in the two sub-operations to a first computing engine, wherein the first sub-operation generates a first portion of a final output of the tensor operation;
assigning a second sub-operation in the two sub-operations to a second computing engine, wherein the second sub-operation generates a second portion of the final output of the tensor operation; and
generating instructions for performing, in parallel, the first sub-operation by the first computing engine and the second sub-operation by the second computing engine.
20. The non-transitory computer-readable medium of claim 19 , wherein the instructions further cause the one or more processors to perform operations including:
generating instructions for making an inference based on a result of the first sub-operation, a result of the second sub-operation, or both,
wherein the first computing engine and the second computing engine are in a same integrated circuit device or in two different integrated circuit devices.