System and method of determining and executing deep tensor columns in neural networks
A system and method may partition and/or execute a NN, by, for a graph including nodes and hyper edges, each node representing a data item in the NN and each hyper edge representing an operation in the NN, identifying a deep tensor column comprising a subset of the nodes and a subset of the hyper edges, such that the operations in the deep tensor column, when executed, use only data which fits within a preselected cache.
1. A method of partitioning a neural network (NN), the method comprising:
receiving a representation of a NN comprising nodes and edges; and
identifying a deep tensor column comprising a subset of the nodes and a subset of the edges, such that the operations in the deep tensor column, when executed, use only data which fits within a preselected cache and wherein identifying a deep tensor column comprises using a cost function measured by the compute to memory ratio of the deep tensor column, the compute to memory ratio determining a measure of the ratio of compute to memory accesses.
2. The method of claim 1 , where data fits within a preselected cache when, during execution in a processor according to the processor's cache policy, it is expected that a cache eviction is not expected until the end of the execution of the task.
3. The method of claim 1 , wherein partitioning comprises iteratively adding a data item in the NN to a deep tensor column until the adding of a data item causes the deep tensor column to have a cost function exceeding a threshold.
4. The method of claim 1 , wherein partitioning comprises iteratively adding a data item in the NN to a deep tensor column and minimizing the cost of a set of component partitions comprised in the deep tensor column.
5. The method of claim 1 , comprising identifying a plurality of deep tensor columns, and executing the plurality of deep tensor columns in order to execute the NN.
6. The method of claim 1 , wherein the deep tensor column comprises a set of elemental operations which form a portion of a layer of the NN.
7. The method of claim 1 , wherein identifying a deep tensor column comprises adding an operation to a deep tensor column if the compute to memory ratio of the operation is below a threshold.
8. A system for partitioning a neural network (NN), the system comprising:
a memory; and
a processor to:
receive a NN comprising nodes and edges; and
identify a deep tensor column comprising a subset of the nodes and a subset of the edges, such that the operations in the deep tensor column, when executed, use only data which fits within a preselected cache; wherein identifying a deep tensor column comprises using a cost function measured by the compute to memory ratio of the deep tensor column, the compute to memory ratio determining a measure of the ratio of compute to memory accesses.
9. The system of claim 8 , where data fits within a preselected cache when, during execution in a processor according to the processor's cache policy, it is expected that a cache eviction is not expected until the end of the execution of the task.
10. The system of claim 8 , wherein partitioning comprises iteratively adding a data item in the NN to a deep tensor column until the adding of a data item causes the deep tensor column to have a cost function exceeding a threshold.
11. The system of claim 8 , wherein partitioning comprises iteratively adding a data item in the NN to a deep tensor column and minimizing the cost of a set of component deep tensor columns comprised in the deep tensor column.
12. The system of claim 8 , wherein the processor is to identify a plurality of deep tensor columns, and the plurality of deep tensor columns are executed in order to execute the NN.
13. The system of claim 8 , wherein the deep tensor column comprises a set of elemental operations which form a portion of a layer of the NN.
14. The system of claim 8 , wherein identifying a deep tensor column comprises adding an operation to a deep tensor column if the compute to memory ratio of the operation is below a threshold.
15. A method of executing a neural network (NN), the method comprising:
receiving a set NN partitions, each NN partition comprising nodes and edges, such that the operations in each partition, when executed, use only data which fits within a preselected cache, wherein the NN partitions are identified using a cost function measured by the compute to memory ratio of a partition, the compute to memory ratio determining a measure of the ratio of compute to memory accesses; and
executing the plurality of NN partitions in order to execute the NN.
16. The method of claim 15 , where data fits within a preselected cache when, during execution in a processor according to the processor's cache policy, it is expected that a cache eviction is not expected until the end of the execution of the task.
17. The method of claim 15 , comprising partitioning the NN by iteratively adding a data item in the NN to a partition until the adding of a data item causes the partition to have a cost function exceeding a threshold.
18. The method of claim 15 , comprising partitioning the NN by iteratively adding a data item in the NN to partition and minimizing the cost of a set of component partitions comprised in the deep tensor column.
19. The method of claim 15 , wherein the partition comprises a set of elemental operations which form a portion of a layer of the NN.
20. The method of claim 15 , wherein identifying a partition comprises adding an operation to a partition if the compute to memory ratio of the operation is below a threshold.