Efficient utilization of processing element array
Generating instructions for programming a processing element array to implement a convolution operation can include determining that the convolution operation under-utilizes the processing element array. The convolution operation involves using the processing element array to perform a series of matrix multiplications between a set of filters and a set of input matrices. Each filter comprises a weight matrix. Each input matrix is assigned to a respective row in the processing element array. Under-utilization can be determined through detecting that less than a threshold number of rows would be used concurrently. In response to determining that the convolution operation under-utilizes the processing element array, instructions can be added for modifying the convolution operation to increase the number of rows used concurrently. The added instructions are executable to cause at least one input matrix to be processed in parallel across more rows compared to processing without modifying the convolution operation.
1. A computer-implemented method performed by one or more processors of a computer system, the computer-implemented method comprising:
determining, by a compiler executing on the one or more processors and as part of generating instructions for programming a processing element array to implement a convolution operation in a neural network, that the convolution operation under-utilizes the processing element array, wherein:
the convolution operation involves using the processing element array to perform a series of matrix multiplications between a set of filters and a set of input matrices,
each filter in the set of filters comprises a weight matrix to be multiplied with a respective input matrix in the set of input matrices,
each input matrix is assigned to a respective row in the processing element array, and
determining that the convolution operation under-utilizes the processing element array comprises detecting that less than a threshold number of rows in the processing element array would be used concurrently; and
adding, by the compiler in response to determining that the convolution operation under-utilizes the processing element array, instructions for modifying the convolution operation to increase the number of rows used concurrently, wherein the added instructions are executable to cause a first input matrix to be processed in parallel across a plurality of rows, the plurality of rows having at least one additional row compared to when the first input matrix is processed without modifying the convolution operation.
2. The computer-implemented method of claim 1 , wherein the added instructions are configured to increase the number of rows for a plurality of input matrices.
3. The computer-implemented method of claim 1 , wherein the added instructions include instructions for:
loading different filter elements from a first filter into the plurality of rows; and
replicating the first input matrix at least once across the plurality of rows.
4. The computer-implemented method of claim 1 , wherein the added instructions include instructions for:
padding the first input matrix using padding data, such that each row of the plurality of rows receives a same number of input elements for processing.
5. The computer-implemented method of claim 4 , wherein the padding data for a first row of the plurality of rows comprises data values from the first input matrix, and wherein the data values from the first input matrix are replicated as non-padding data for a second row subsequent to the first row.
6. The computer-implemented method of claim 4 , wherein the padding data includes data used for memory alignment, and wherein the added instructions include instructions for discarding results computed from the data used for memory alignment.
7. The computer-implemented method of claim 1 , wherein the added instructions include instructions for:
loading a first filter element from a first filter into a first row of the plurality of rows;
loading a second filter element from the first filter into a second row of the plurality of rows;
dividing the first input matrix into at least a first partition and a second partition;
inputting the first partition to the first row for processing using the first filter element; and
inputting the second partition to the second row for processing using the second filter element, in parallel with the processing of the first partition.
8. The computer-implemented method of claim 7 , wherein the inputting of the first partition and the second partition comprises:
reading the first partition and the second partition from respective memory locations;
forming a first sequential stream of input elements corresponding to the first partition; and
forming a second sequential stream of input elements corresponding to the second partition.
9. The computer-implemented method of claim 7 , wherein the added instructions include instructions for:
loading a third filter element from the first filter into a third row of the plurality of rows; and
replicating the first partition to form input to the third row, wherein input elements of the third row are data shifted relative to input elements of the first row.
10. A computer system, comprising:
a processing element array;
one or more processors separate from the processing element array; and
a memory subsystem storing a compiler, the compiler being executable by the one or more processors to generate instructions for programming the processing element array to implement a convolution operation in a neural network, wherein:
the convolution operation involves using the processing element array to perform a series of matrix multiplications between a set of filters and a set of input matrices,
each filter in the set of filters comprises a weight matrix to be multiplied with a respective input matrix in the set of input matrices, and
each input matrix is assigned to a respective row in the processing element array; and
wherein the compiler is configured to generate the instructions for programming the processing element array through:
determining that the convolution operation under-utilizes the processing element array, based on detecting that less than a threshold number of rows in the processing element array would be used concurrently, and
responsive to determining that the convolution operation under-utilizes the processing element array, adding instructions for modifying the convolution operation to increase the number of rows used concurrently, wherein the added instructions are executable to cause a first input matrix to be processed in parallel across a plurality of rows, the plurality of rows having at least one additional row compared to when the first input matrix is processed without modifying the convolution operation.
11. The computer system of claim 10 , wherein the added instructions are configured to increase the number of rows for a plurality of input matrices.
12. The computer system of claim 10 , wherein the added instructions include instructions for:
loading different filter elements from a first filter into the plurality of rows; and
replicating the first input matrix at least once across the plurality of rows.
13. The computer system of claim 10 , wherein the added instructions include instructions for:
padding the first input matrix using padding data, such that each row of the plurality of rows receives a same number of input elements for processing.
14. The computer system of claim 13 , wherein the padding data for a first row of the plurality of rows comprises data values from the first input matrix, and wherein the data values from the first input matrix are replicated as non-padding data for a second row subsequent to the first row.
15. The computer system of claim 13 , wherein the padding data includes data used for memory alignment, and wherein the added instructions include instructions for discarding results computed from the data used for memory alignment.
16. The computer system of claim 10 , wherein the added instructions include instructions for:
loading a first filter element from a first filter into a first row of the plurality of rows;
loading a second filter element from the first filter into a second row of the plurality of rows;
dividing the first input matrix into at least a first partition and a second partition;
inputting the first partition to the first row for processing using the first filter element; and
inputting the second partition to the second row for processing using the second filter element, in parallel with the processing of the first partition.
17. The computer system of claim 16 , wherein the inputting of the first partition and the second partition comprises:
reading the first partition and the second partition from respective memory locations;
forming a first sequential stream of input elements corresponding to the first partition; and
forming a second sequential stream of input elements corresponding to the second partition.
18. The computer system of claim 16 , wherein the added instructions include instructions for:
loading a third filter element from the first filter into a third row of the plurality of rows; and
replicating the first partition to form input to the third row, wherein input elements of the third row are data shifted relative to input elements of the first row.
19. A non-transitory computer readable medium having stored therein instructions that, when executed by one or more processors, cause the one or more processors to execute a compiler, the compiler performing operations including:
determining, as part of generating instructions for programming a processing element array to implement a convolution operation in a neural network, that the convolution operation under-utilizes the processing element array, wherein:
the convolution operation involves using the processing element array to perform a series of matrix multiplications between a set of filters and a set of input matrices,
each filter in the set of filters comprises a weight matrix to be multiplied with a respective input matrix in the set of input matrices,
each input matrix is assigned to a respective row in the processing element array, and
determining that the convolution operation under-utilizes the processing element array comprises detecting that less than a threshold number of rows in the processing element array would be used concurrently; and
responsive to determining that the convolution operation under-utilizes the processing element array, adding instructions for modifying the convolution operation to increase the number of rows used concurrently, wherein the added instructions are executable to cause a first input matrix to be processed in parallel across a plurality of rows, the plurality of rows having at least one additional row compared to when the first input matrix is processed without modifying the convolution operation.
20. The non-transitory computer readable medium of claim 19 , wherein the added instructions include instructions for:
loading different filter elements from a first filter into the plurality of rows; and
performing one of:
replicating the first input matrix at least once across the plurality of rows, or
partitioning the first input matrix across the plurality of rows.