Systems and methods for neural network convolutional layer matrix multiplication using cache memory
A computer processor may include a number of cores, a shared cache shared among the cores, and a local cache associated with each core and used by that core only. Input data for a neural network (NN) layer may be partitioned into a set of tiles of size T×T, and the tile set may be partitioned into blocks of R tiles. For each block, a core may perform a transform operation on the tiles to produce transformed data matrices fitting in a local cache, and a set of multiply operations, each multiply operation using a transformed data matrix and a transformed kernel matrix from a set of transformed kernel matrices. The set of transformed kernel matrices may fit in the shared cache. The result of at least one of the multiply operations may be stored in a location used to store a transformed data matrix.
1. A method of executing matrix multiply operations for a neural network (NN), the method comprising, using a computer processor comprising a plurality of cores and a shared cache shared among the cores, each core associated with a local cache used by that core only:
partitioning input data for a NN layer into a set of tiles using a parameter T, each tile being of size T×T;
using a parameter R, partitioning the set of tiles into blocks of R tiles each; and
for each block of R tiles, performing by a single core:
a transform operation on the R tiles to produce a set of transformed data matrices, the set of transformed data matrices stored in the local cache of the single core; and
a set of multiply operations, each multiply operation using a transformed data matrix of the set of transformed data matrices and a transformed kernel matrix from a set of transformed kernel matrices, the set of transformed kernel matrices stored in the shared cache, wherein a result of at least one of the multiply operations is stored in a location used to store a transformed data matrix.
2. The method of claim 1 , wherein the partitioning of the input data and the partitioning of the tile set increase a likelihood that the transformed matrices are stored in a local cache.
3. The method of claim 1 , wherein the location used to store the result of at least one of the multiply operations is a location used to store a transformed data matrix used in a previous multiply operation.
4. The method of claim 1 , wherein the set of transformed kernel matrices is an entire set of transformed kernel matrices for the NN layer.
5. The method of claim 1 , comprising performing by the core an inverse transform operation on the results of the multiply operations.
6. The method of claim 5 , comprising, for a second NN layer:
partitioning a portion of the results of the inverse transform operations into a second set of tiles using a parameter T′, each tile being of size T′×T′;
using a parameter R′, partitioning the second set of tiles into blocks of R′ tiles each; and
for each block of R′ tiles, performing by a core a set of transform-multiply-inverse transform operations.
7. The method of claim 1 , wherein the transformed data matrices for the NN layer fit within the shared cache.
8. A system for of executing matrix multiply operations for a neural network (NN), the system comprising a computer processor comprising:
a plurality of cores each core associated with a local cache used by that core only; and
a shared cache shared among the cores, wherein, for input data partitioned for a NN layer into a set of tiles using a parameter T, each tile being of size T×T and the set of tiles partitioned using a parameter R into blocks of R tiles each:
each single core configured to, for a block of R tiles perform:
a transform operation on the R tiles to produce a set of transformed data matrices, the set of transformed data matrices stored in the local cache of the single core; and
a set of multiply operations, each multiply operation using a transformed data matrix of the set of transformed data matrices and a transformed kernel matrix from a set of transformed kernel matrices, the set of transformed kernel matrices stored in the shared cache, wherein a result of at least one of the multiply operations is stored in a location used to store a transformed data matrix.
9. The system of claim 8 , wherein the partitioning of the input data and the partitioning of the tile set increase a likelihood that the transformed matrices are stored in a local cache.
10. The system of claim 8 , wherein the location used to store the result of at least one of the multiply operations is a location used to store a transformed data matrix used in a previous multiply operation.
11. The system of claim 8 , wherein the set of transformed kernel matrices is an entire set of transformed kernel matrices for the NN layer.
12. The system of claim 8 , wherein the core is configured to perform an inverse transform operation on the results of the multiply operations.
13. The system of claim 12 , wherein for a second NN layer a portion of the results of the inverse transform operations is partitioned into a second set of tiles using a parameter T′, each tile being of size T′×T′, and the second set of tiles is partitioned using a parameter R′ into blocks of R′ tiles each; and
each single core is configured to, for a block of R′ tiles, perform a set of transform-multiply-inverse transform operations.
14. The system of claim 8 , wherein the transformed data matrices for the NN layer fit within the shared cache.
15. A method of performing inference for a neural network (NN), the method comprising, in a computer processor comprising a plurality of cores, a shared cache, and a private cache used by each core, the method comprising:
using input data divided into a set of tiles using a parameter T, each tile being of size T×T, the set of tiles divided into blocks of tiles, for each block of tiles, a single core executing a transform-multiply-inverse transform operation on the block of tiles, such that a set of transformed data matrices is stored in the local cache of the single core, wherein the transform-multiply-inverse transform operation operates on a transformed kernel matrix, the transformed kernel matrix stored in the shared cache, and wherein results of a multiply operation is stored in a location used to store a transformed data matrix.
16. The method of claim 15 , wherein the division of the input data and the division of the tile set increase a likelihood that the transformed matrices are stored in a local cache.
17. The method of claim 15 , wherein the location used to store the result is a location used to store a transformed data matrix used in a previous multiply operation.
18. The method of claim 15 , wherein, for a second NN layer a portion of the results of the inverse transform operations are partitioned into a second set of tiles using a parameter T′, each tile being of size T′×T′, and wherein the second set of tiles is partitioned using a parameter R′ into blocks of R′ tiles each, wherein the method comprises for each block of R′ tiles, performing by a core a set of transform-multiply-inverse transform operations.
19. The method of claim 15 wherein in the transform-multiply-inverse operation a set of transformed kernel matrices used in the transform-multiply-inverse operation fits in the shared cache.