Sparsity uniformity enforcement for multicore processor
Methods and systems relating to the field of parallel computing are disclosed herein. The methods and systems disclosed include approaches for sparsity uniformity enforcement for a set of computational nodes which are used to execute a complex computation. A disclosed method includes determining a sparsity distribution in a set of operand data, and generating, using a compiler, a set of instructions for executing, using the set of operand data and a set of processing cores, a complex computation. Alternatively, the method includes altering the operand data. The method also includes distributing the set of operand data to the set of processing cores for use in executing the complex computation in accordance with the set of instructions. Either the altering is conducted to, or the compiler is programmed to, balance the sparsity distribution among the set of processing cores.
1. A computer-implemented method comprising:
altering a set of operand data to increase a degree of sparsity in a sparsity distribution of the set of operand data, wherein the set of operand data is for a complex computation; and
distributing the set of operand data to a set of processing cores for executing the complex computation;
wherein the altering balances the sparsity distribution among the set of processing cores.
2. The computer-implemented method of claim 1 , further comprising:
generating, using a compiler, a set of instructions for executing, using the set of operand data and a set of processing cores, the complex computation;
wherein the altering is conducted by the compiler prior to the generating of the set of instructions.
3. The computer-implemented method of claim 1 , further comprising:
generating, using a compiler, a set of instructions for executing, using the set of operand data and a set of processing cores, the complex computation;
wherein the altering is conducted by the compiler after assigning the set of instructions to the set of processing cores.
4. The computer-implemented method of claim 1 , wherein:
the altering is conducted on the set of processing cores and is based on the distributing of the set of operand data.
5. The computer-implemented method of claim 1 , wherein:
the complex computation is an execution of an artificial neural network;
the set of operand data is execution data; and
the altering is conducted on the set of processing cores and is based on the distributing of the set of operand data in that the processing cores in the set of processing cores each have a partial view of the distributing of the set of operand data.
6. The computer-implemented method of claim 1 , wherein:
the complex computation is an execution of an artificial neural network; and
the set of operand data is network data.
7. The computer-implemented method of claim 6 , further comprising:
reversing the alteration of the set of operand data after executing the complex computation with a first input; and
realtering the set of operand data to increase the degree of sparsity of the set of operand data prior to a second execution of the complex computation with a second input.
8. The computer-implemented method of claim 7 , wherein:
the execution of the complex computation with the first input and the execution of the complex computation with the second input are conducted while training the artificial neural network.
9. The computer-implemented method of claim 1 , wherein:
the complex computation is an execution of an artificial neural network;
the set of operand data is network data of the artificial neural network; and
the altering includes randomly selecting, prior to executing the complex computation, data entries from the set of operand data to alter to a non-sparse value.
10. The computer-implemented method of claim 1 , wherein:
the complex computation is an execution of an artificial neural network;
the set of operand data is execution data; and
the altering includes randomly selecting, while executing the complex computation, data entries from the set of operand data to alter to a non-sparse value.
11. The computer-implemented method of claim 1 , wherein:
the complex computation is an execution of an artificial neural network;
the set of operand data is execution data; and
the altering includes obtaining an estimate of a computation using the execution data and altering, based on the estimate, data entries from the set of operand data to alter to a non-sparse value.
12. The computer-implemented method of claim 11 , wherein:
the estimate of the computation is obtained using the execution data.
13. The computer-implemented method of claim 1 , wherein:
the set of operand data includes sparse values and non-sparse values; and
the set of operand data is distributed to the processing cores in the set of processing cores in a format which includes geometric information provided by the non-sparse values of the operand data in the set of operand data.
14. The computer-implemented method of claim 1 , wherein:
the set of operand data is organized into a set of blocks;
the complex computation is an operation on a first tensor and a second tensor;
the blocks in the set of blocks are each larger than a single data entry; and
the blocks in the set of blocks are each smaller than the first tensor and the second tensor.
15. The computer-implemented method of claim 1 , wherein:
the set of operand data is organized into a set of blocks; and
the altering of the set of operand data is conducted so that the sparsity distribution is applied evenly across the set of blocks.
16. A system comprising:
a set of processing cores;
at least one controller programmed to alter a set of operand data to increase a degree of sparsity in a sparsity distribution of the set of operand data, wherein the set of operand data is for a complex computation;
a network programmed to distribute the set of operand data to the set of processing cores for use in executing the complex computation; and
whereby the altering balances the sparsity distribution among the set of processing cores.
17. The system of claim 16 , wherein:
the at least one controller is an external controller with a compiler to alter the set of operand data prior to the execution of the complex computation.
18. The system of claim 16 , wherein:
the at least one controller includes a set of local controllers on the set of processing cores in a one-to-one correspondence; and
the set of local controllers alter the set of operand data to increase the degree of sparsity during execution of the complex computation.
19. The system of claim 16 , further comprising:
a random access memory shared by the set of processing cores;
wherein the network distributes the set of operand data to the set of processing cores from the random access memory.
20. A system comprising:
a set of processing cores;
a means for altering a set of operand data to increase a degree of sparsity in a sparsity distribution of the set of operand data, wherein the set of operand data is for a complex computation; and
a means for distributing the set of operand data to a set of processing cores for executing the complex computation;
wherein the means for altering balances the sparsity distribution among the set of processing cores.