Sparsity uniformity enforcement for multicore processor
Methods and systems relating to the field of parallel computing are disclosed herein. The methods and systems disclosed include approaches for sparsity uniformity enforcement for a set of computational nodes which are used to execute a complex computation. A disclosed method includes determining a sparsity distribution in a set of operand data, and generating, using a compiler, a set of instructions for executing, using the set of operand data and a set of processing cores, a complex computation. Alternatively, the method includes altering the operand data. The method also includes distributing the set of operand data to the set of processing cores for use in executing the complex computation in accordance with the set of instructions. Either the altering is conducted to, or the compiler is programmed to, balance the sparsity distribution among the set of processing cores.
1. A computer-implemented method comprising:
determining a sparsity distribution in a set of operand data;
generating, using a compiler, a set of instructions for executing, using the set of operand data and a set of processing cores, a complex computation; and
distributing the set of operand data to the set of processing cores for use in executing the complex computation in accordance with the set of instructions;
wherein the compiler is programmed to utilize the sparsity distribution in the set of operand data, when generating the set of instructions, to balance the sparsity distribution among the set of processing cores.
2. The computer-implemented method of claim 1 , wherein:
the determining of the sparsity distribution is conducted by evaluating the set of operand data and measuring the sparsity distribution; and
the sparsity distribution is a property of the set of operand data prior to the determining of the sparsity distribution.
3. The computer-implemented method of claim 1 , wherein:
the determining of the sparsity distribution is conducted by altering the set of operand data; and
the sparsity distribution is a property of the set of operand data produced by the determining of the sparsity distribution.
4. The computer-implemented method of claim 1 , wherein:
the determining of the sparsity distribution is conducted by generating a set of sparsity inducing commands to alter the set of operand data during execution of the set of instructions; and
the sparsity distribution is a property of the set of operand data produced during the execution of the set of instructions.
5. The computer-implemented method of claim 1 , wherein:
the set of processing cores are designed to operate with an instruction set;
a target language of the compiler is the instruction set; and
a source language of the compiler is source code.
6. The computer-implemented method of claim 5 , wherein:
a source code description of the complex computation includes an operation;
at least two instructions in the set of instructions are required to execute the operation; and
the compiler is programmed to utilize the sparsity distribution in the set of operand data when defining operands for the at least two instructions.
7. The computer-implemented method of claim 1 , wherein:
the compiler is programmed to utilize the sparsity distribution in the set of operand data when placing instructions from the set of instructions with processing cores in the set of processing cores.
8. The computer-implemented method of claim 1 , wherein:
the compiler is programmed to utilize the sparsity distribution in the set of operand data when the set of instructions are executed by the processing cores in the set of processing cores.
9. The computer-implemented method of claim 1 , wherein:
the complex computation is an execution of an artificial neural network;
the set of operand data includes network data and execution data; and
the compiler is programmed to utilize the sparsity distribution in the network data and execution data, when generating the set of instructions, to balance the sparsity distribution among the set of processing cores.
10. The computer-implemented method of claim 1 , wherein:
the complex computation is an execution of an artificial neural network;
the set of operand data includes network data;
the determining of the sparsity distribution is conducted by altering the set of operand data; and
the sparsity distribution is a property of the set of operand data produced by the determining of the sparsity distribution.
11. The computer-implemented method of claim 1 , wherein:
the complex computation is an execution of an artificial neural network;
the set of operand data includes network data and execution data;
the determining of the sparsity distribution is conducted by generating a set of sparsity inducing commands to alter the set of operand data during execution of the set of instructions; and
the sparsity distribution is a property of the network data and execution data produced by the determining of the sparsity distribution.
12. The computer-implemented method of claim 1 , wherein:
the set of operand data is distributed to the processing cores in the set of processing cores in a format which includes geometric information provided by non-sparse values of the set of operand data.
13. The computer-implemented method of claim 1 , wherein:
the generating of the set of instructions includes organize the operand data in the set of operand data into a set of blocks; and
the set of blocks are a set of operands for the set of instructions.
14. The computer-implemented method of claim 13 , wherein:
the complex computation is an operation on a first tensor and a second tensor;
at least two instructions in the set of instructions are required to execute the operation;
the blocks in the set of blocks are each larger than a single data entry; and
the blocks in the set of blocks are each smaller than the first tensor and the second tensor.
15. The computer-implemented method of claim 13 , wherein:
the determining of the sparsity distribution is conducted by altering the set of operand data so that the sparsity distribution is applied evenly across the set of blocks.
16. A system comprising:
a set of processing cores;
a compiler programmed to: (i) determine a sparsity distribution for of a set of operand data; and (ii) generate a set of instructions for executing a complex computation using the set of operand data and the set of processing cores; and
a network programmed to distribute the set of operand data to the set of processing cores for use in executing the complex computation in accordance with the set of instructions;
wherein the compiler is further programmed to utilize the sparsity distribution in the set of operand data, when generating the set of instructions, to balance the sparsity distribution among the set of processing cores.
17. The system of claim 16 , further comprising:
a substrate for the set of processing cores;
wherein the network is programmed by the compiler; and
wherein the compiler is instantiated by a processor on the substrate.
18. The system of claim 16 , further comprising:
a random access memory shared by the set of processing cores;
wherein the network distributes the set of operand data to the set of processing cores from the random access memory.
19. The system of claim 16 , further comprising:
an external memory; and
an external controller;
wherein the compiler is instantiated by the external controller; and
wherein the network distributes the set of operand data to the set of processing cores from the external memory.
20. A system comprising:
a means for determining a sparsity distribution in a set of operand data;
a means for generating a set of instructions for executing a complex computation using the set of operand data and a set of processing cores; and
a means for distributing the set of operand data to the set of processing cores for use in executing the complex computation in accordance with the set of instructions;
wherein the means for generating utilizes the sparsity distribution in the set of operand data, when generating the set of instructions, to balance the sparsity distribution among the set of processing cores.