Sparsity uniformity enforcement for multicore processor
Methods and systems relating to the field of parallel computing are disclosed herein. The methods and systems disclosed include approaches for sparsity uniformity enforcement for a set of computational nodes which are used to execute a complex computation. A disclosed method includes determining a sparsity distribution in a set of operand data, and generating, using a compiler, a set of instructions for executing, using the set of operand data and a set of processing cores, a complex computation. Alternatively, the method includes altering the operand data. The method also includes distributing the set of operand data to the set of processing cores for use in executing the complex computation in accordance with the set of instructions. Either the altering is conducted to, or the compiler is programmed to, balance the sparsity distribution among the set of processing cores.
1. A computer-implemented method comprising:
altering a set of operand data to increase a degree of sparsity in a sparsity distribution of the set of operand data, wherein the set of operand data is for a complex computation; and
distributing the set of operand data to a set of computational nodes for executing the complex computation;
wherein the altering balances the sparsity distribution among the set of computational nodes.
2. The computer-implemented method of claim 1 , wherein:
the set of operand data is distributed to the set of computational nodes from a shared memory of the set of computational nodes.
3. The computer-implemented method of claim 2 , wherein:
the set of computational nodes is a set of processing cores;
the shared memory is on a same substrate as the set of computational nodes; and
the set of computational nodes are in a wafer-scale single chip system.
4. The computer-implemented method of claim 2 , wherein:
the complex computation is an artificial neural network; and
the set of operand data is a set of network data of the artificial neural network.
5. The computer-implemented method of claim 4 , wherein:
the set of operand data is distributed in a set of blocks where each block in the set of blocks holds a portion of operand data; and
the portion of operand data is larger than a single value and smaller than a layer of the artificial neural network from which the portion of operand data was taken.
6. The computer-implemented method of claim 1 , further comprising:
compressing the set of operand data after altering the set of operand data and prior to distributing the set of operand data;
whereby the set of operand data is distributed to the set of computational nodes in a compressed form.
7. The computer-implemented method of claim 6 , wherein:
the complex computation is the execution of an artificial neural network;
the artificial neural network is a natural language processing artificial neural network;
a set of non-sparse values in the set of operand data provide geometric information regarding a set of sparse values in the set of operand data; and
the compressed form of the set of operand data preserves the geometric information of the set of operand data.
8. The computer-implemented method of claim 1 , wherein:
the distributing of the set of operand data includes programming a network of the set of computational nodes for the distributing of the set of operand data.
9. The computer-implemented method of claim 1 , wherein:
the set of operand data is distributed to the set of computational nodes from a shared memory of the set of computational nodes; and
the altering of the set of operand data is conducted by a hardware module in association with the shared memory.
10. The computer-implemented method of claim 1 , wherein:
the altering of the set of operand data is conducted by network components of the set of computational nodes as the set of operand data is transmitted between computational nodes in the set of computational nodes.
11. The computer-implemented method of claim 1 , wherein:
the altering of the set of operand data is conducted by setting values below a specific magnitude to zero.
12. The computer-implemented method of claim 1 , wherein:
the complex computation is an execution of an artificial neural network during training of the artificial neural network;
the set of operand data is network data of the artificial neural network; and
the method comprises retaining a set of values of the set of operand data prior to the altering.
13. A computer-implemented method comprising:
determining a sparsity distribution in a set of operand data;
generating, using a compiler, a set of instructions for executing, using the set of operand data and a set of computational nodes, a complex computation; and
distributing the set of operand data to the set of computational nodes for use in executing the complex computation in accordance with the set of instructions;
wherein the compiler is programmed to utilize the sparsity distribution in the set of operand data, when generating the set of instructions, to balance the sparsity distribution among the set of computational nodes.
14. The computer-implemented method of claim 13 , wherein:
the set of operand data is distributed to the set of computational nodes from a shared memory of the set of computational nodes.
15. The computer-implemented method of claim 14 , wherein:
the set of computational nodes is a set of processing cores;
the shared memory is on a same substrate as the set of computational nodes; and
the set of computational nodes are in a wafer-scale single chip system.
16. The computer-implemented method of claim 13 , wherein:
the complex computation is an artificial neural network; and
the set of operand data is network data of the artificial neural network.
17. The computer-implemented method of claim 16 , wherein:
the set of operand data is distributed in a set of blocks where each block in the set of blocks holds a portion of operand data; and
the portion of operand data is larger than a single value and smaller than a layer of the artificial neural network from which the portion of operand data was taken.
18. The computer-implemented method of claim 13 , further comprising:
compressing the set of operand data prior to distributing the set of operand data;
whereby the set of operand data is distributed to the set of computational nodes in a compressed form.
19. The computer-implemented method of claim 18 , wherein:
the complex computation is an execution of an artificial neural network;
the artificial neural network is a natural language processing artificial neural network;
a set of non-sparse values in the set of operand data provide geometric information regarding a set of sparse values in the set of operand data; and
the compressed form of the set of operand data preserves the geometric information of the set of operand data.
20. The computer-implemented method of claim 13 , wherein:
the distributing of the set of operand data includes programming a network of the set of computational nodes for the distributing of the set of operand data.
21. The computer-implemented method of claim 13 , wherein:
the set of operand data is distributed to the set of computational nodes from a shared memory of the set of computational nodes; and
the determining of the sparsity distribution of the set of operand data is conducted by a hardware module in association with the shared memory.
22. A system comprising:
a set of computational nodes, in the form of a set of processing cores;
at least one controller programmed to alter a set of operand data to increase a degree of sparsity in a sparsity distribution of the set of operand data, wherein the set of operand data is for a complex computation;
a network programmed to distribute the set of operand data to the set of computational nodes for use in executing the complex computation; and
whereby the altering balances the sparsity distribution among the set of computational nodes.
23. The system of claim 22 , further comprising:
a shared memory of the set of computational nodes;
wherein the set of operand data is distributed to the set of computational nodes from the shared memory of the set of computational nodes.
24. The system of claim 23 , wherein:
the shared memory is on a same substrate as the set of computational nodes; and
the system is a wafer-scale single chip system.
25. The system of claim 22 , wherein:
the complex computation is an artificial neural network; and
the set of operand data is network data of the artificial neural network.
26. The system of claim 25 , wherein:
the set of operand data is distributed in a set of blocks where each block in the set of blocks holds a portion of operand data; and
the portion of operand data is larger than a single value and smaller than a layer of the artificial neural network from which the portion of operand data was taken.
27. The system of claim 22 , wherein:
the complex computation is an execution of an artificial neural network;
the artificial neural network is a natural language processing artificial neural network;
the set of operand data is compressed into a compressed form prior to being distributed to the set of computational nodes;
a set of non-sparse values in the set of operand data provide geometric information regarding a set of sparse values in the set of operand data; and
the compressed form of the set of operand data preserves the geometric information of the set of operand data.
28. The system of claim 22 , wherein:
the distributing of the set of operand data includes programming a network of the set of computational nodes for the distributing of the set of operand data.
29. The system of claim 22 , further comprising:
a shared memory of the set of computational nodes; and
a hardware module;
wherein: the set of operand data is distributed to the set of computational nodes from the shared memory of the set of computational nodes; and the altering of the set of operand data is conducted by the hardware module in association with the shared memory.
30. The system of claim 22 , further comprising:
network components of the set of computational nodes;
wherein the altering of the set of operand data is conducted by the network components of the set of computational nodes as the set of operand data is transmitted between computational nodes in the set of computational nodes.
31. The system of claim 22 , wherein:
the altering of the set of operand data is conducted by setting values below a specific magnitude to zero.
32. The system of claim 22 , wherein:
the complex computation is an execution of an artificial neural network during training of the artificial neural network;
the set of operand data is network data of the artificial neural network; and
the system retains a set of values of the set of operand data prior to the altering.
33. A system comprising:
a set of computational nodes, in the form of a set of processing cores;
a compiler instantiated by a processor and programmed to: (i) determine a sparsity distribution for of a set of operand data; and (ii) generate a set of instructions for executing a complex computation using the set of operand data and the set of computational nodes; and
a network programmed to distribute the set of operand data to the set of computational nodes for use in executing the complex computation in accordance with the set of instructions;
wherein the compiler is further programmed to utilize the sparsity distribution in the set of operand data, when generating the set of instructions, to balance the sparsity distribution among the set of computational nodes.
34. The system of claim 33 , further comprising:
a shared memory of the set of computational nodes;
wherein the set of operand data is distributed to the set of computational nodes from the shared memory of the set of computational nodes.
35. The system of claim 34 , wherein:
the shared memory is on a same substrate as the set of computational nodes; and
the system is a wafer-scale single chip system.
36. The system of claim 33 , wherein:
the complex computation is an artificial neural network; and
the set of operand data is network data of the artificial neural network.
37. The system of claim 36 , wherein:
the set of operand data is distributed in a set of blocks where each block in the set of blocks holds a portion of operand data; and
the portion of operand data is larger than a single value and smaller than a layer of the artificial neural network from which the portion of operand data was taken.
38. The system of claim 33 , wherein:
the complex computation is an execution of an artificial neural network;
the artificial neural network is a natural language processing artificial neural network;
the set of operand data is compressed into a compressed form prior to being distributed to the set of computational nodes;
a set of non-sparse values in the set of operand data provide geometric information regarding a set of sparse values in the set of operand data; and
the compressed form of the set of operand data preserves the geometric information of the set of operand data.
39. The system of claim 33 , wherein:
the distributing of the set of operand data includes programming a network of the set of computational nodes for the distributing of the set of operand data.
40. The system of claim 33 , further comprising:
a shared memory of the set of computational nodes; and
a hardware module;
wherein: the set of operand data is distributed to the set of computational nodes from the shared memory of the set of computational nodes; and the determining of the sparsity distribution of the set of operand data is conducted by the hardware module in association with the shared memory.