Increasing sparcity in data sets
Apparatuses, systems, and techniques to transform data sets, such as matrices representing layers of neural networks, to increase sparsity and/or other characteristics of said data sets to improve performance in computations, such as neural network computations. In at least one embodiment, one or more subsets of data in one or more sets of data are rearranged as part of a process to increase sparsity in said one or more sets of data to satisfy one or more one or more structural sparsity constraints.
1. One or more processors, comprising: circuitry to cause an increase in sparsity of one or more sets of data by at least rearranging elements in the one or more sets of data, such that the one or more sets of data comprise one or more subsets of data conforming to one or more structural sparsity constraints.
2. The one or more processors of claim 1 , wherein the one or more structural sparsity constraints impose a limitation on the one or more sets of data whereby the one or more subsets of data comprise at least a first quantity of non-zero data values and a second quantity of zero data values.
3. The one or more processors of claim 1 , wherein the one or more circuits rearrange circuitry rearranges elements in the one or more sets of data by swapping a first subset of the one or more subsets of data having a first position in the one or more sets of data with a second subset of the one or more subsets of data having a second position in the one or more sets of data.
4. The one or more processors of claim 1 , wherein:
the one or more sets of data comprise numerical values accessible using at least a first index and a second index; and
the one or more sets of data comprise a first subset of data associated with the first index and a second subset of data associated with the second index; and
the circuitry causes the increase in sparsity by at least exchanging the numerical values of the first subset of data associated with the first index with the second subset of data associated with a second matrix.
5. The one or more processors of claim 1 , wherein the one or more sets of data comprise numerical values corresponding to weight parameters associated with one or more neural networks.
6. The one or more processors of claim 1 , wherein each data value in the one or more sets of data is associated with a first index value and a second index value, the first index value and the second index value indicating a position of each data value in the one or more sets of data.
7. The one or more processors of claim 1 , wherein the circuitry causes the increase in sparsity of the one or more sets of data using a deep learning framework to determine a set of transforms to swap two or more subsets of data in the one or more sets of data, the deep learning framework further setting one or more data values in the two or more subsets of data to zero.
8. The one or more processors of claim 1 , wherein rearranging the elements in the one or more sets of data is performed by permuting the elements in the one or more sets of data.
9. The one or more processors of claim 1 , wherein rearranging the elements in the one or more sets of data is performed by pruning the elements in the one or more sets of data.
10. A system comprising:
one or more processors; and
memory including instructions that, when executed by the one or more processors, cause computer system to at least:
cause an increase in sparsity of one or more sets of data by at least rearranging elements in the one or more sets of data, such that the one or more sets of data comprise one or more subsets of data conforming to one or more structural sparsity constraints.
11. The system of claim 10 , wherein the instructions further include instructions that, when executed by the one or more processors, cause a deep learning framework to transform the one or more sets of data based, at least in part, on the one or more structural sparsity constraints.
12. The system of claim 11 , wherein the one or more structural sparsity constraints comprise at least one structural sparsity constraint wherein a subset of the one or more sets of data comprises at least a first quantity of non-zero data values and a second quantity of zero data values.
13. The system of claim 11 , wherein the deep learning framework transforms the one or more sets of data by exchanging at least a first subset of the one or more sets of data associated with a first position value with at least a second subset of the one or more sets of data associated with a second position value.
14. The system of claim 11 , wherein the deep learning framework determines a set of transforms to rearrange the elements in the one or more sets of data by:
randomly selecting a first subset of the one or more sets of data associated with a first position value and a second subset of the one or more sets of data associated with a second position value;
exchanging the first subset and the second subset;
setting one or more data values in the first subset and the second subset to a zero value;
calculating a metric associated with a neural network corresponding to the one or more sets of data; and
as a result of the metric being greater than another metric, adding a transform comprising the first position value and the second position value to the set of transforms.
15. The system of claim 10 , wherein the one or more sets of data are associated with one or more layers of a neural network and the one or more sets of data values comprise only non-zero numerical values.
16. The system of claim 10 , wherein the instructions further include instructions that, when executed by the one or more processors, cause the increase in sparsity by setting one or more data values in a subset of the one or more sets of data to a zero value, the one or more data values in the subset being numerical values representing one or more weight values associated with a neural network and the subset being determined based, at least in part, on one of the one or more structural sparsity constraints.
17. The system of claim 10 , wherein the one or more processors are parallel processing units, the parallel processing units comprising one or more sparse tensor cores to accelerate one or computations on the one or more sets of data based, at least in part, on the one or more structural sparsity constraints on the one or more sets of data.
18. The system of claim 10 , wherein rearranging the elements in the one or more sets of data is performed by permuting the elements in the one or more sets of data.
19. The system of claim 10 , wherein rearranging the elements in the one or more sets of data is performed by pruning the elements in the one or more sets of data.
20. A method comprising:
causing an increase in sparsity of one or more sets of data by at least rearranging elements in the one or more sets of data, such that the one or more sets of data comprise one or more subsets of data conforming to one or more structural sparsity constraints.
21. The method of claim 20 , further comprising causing the increase in sparsity of the one or more sets of data using a deep learning framework, wherein the deep learning framework at least rearranges the elements in the one or more sets of data.
22. The method of claim 21 , wherein the one or more structural sparsity constraints comprise at least a limitation on the one or more sets of data, the limitation requiring the one or more subsets of data to comprise at least a first quantity of non-zero data values and a second quantity of zero data values.
23. The method of claim 21 , wherein the deep learning framework rearranges the elements in the one or more sets of data by swapping a first subset of the one or more subsets of data having a first position in the one or more sets of data with a second subset of the one or more subsets of data having a second position in the one or more sets of data.
24. The method of claim 21 , wherein the deep learning framework determines a set of transforms to rearrange the elements in the one or more sets of data by:
randomly selecting a first subset of the one or more subsets of data associated with a first position value and a second subset of the one or more subsets of data associated with a second position value;
exchanging the first subset and the second subset;
setting one or more data values in the first subset and the second subset to a zero value;
calculating a metric value associated with a neural network corresponding to the one or more sets of data; and
as a result of the metric value being greater than another metric value, adding a transform comprising the first position value and the second position value to the set of transforms.
25. The method of claim 21 , wherein the deep learning framework determines a set of transforms to rearrange the elements in the one or more sets of data by:
selecting a first subset of the one or more sets of data and a second subset of the one or more sets of data;
determining one or more first transforms of the first subset, the one or more first transforms exchanging a first one or more sets of sequential data values in the first subset with a second one or more sets of sequential data values in the first subset such that a first metric value associated with a neural network is greater than a second metric value associated with the neural network;
determining one or more second transforms of the second subset, the one or more second transforms exchanging a first one or more sets of sequential data values in the second subset with a second one or more sets of sequential data values in the second subset such that a third metric value associated with the neural network is greater than the second metric value associated with the neural network; and
combining the one or more first transforms with the one or more second transforms into the set of transforms.
26. The method of claim 20 , wherein the one or more sets of data are associated with one or more layers of a neural network having an improved metric value as a result of the increase in sparsity.
27. The method of claim 20 , wherein the increase in sparsity is further caused by setting one or more data values in a subset of the one or more sets of data to a zero data value such that one or more operands of one or more computations on the one or more sets of data satisfy the one or more structural sparsity constraints.
28. The method of claim 20 , wherein one or more parallel processing units perform one or more computations on the one or more sets of data, the one or more parallel processing units comprising one or more sparse tensor cores to utilize the one or more structural sparsity constraints as a result of the increase in sparsity.
29. The method of claim 20 , wherein rearranging the elements in the one or more sets of data is performed by permuting the elements in the one or more sets of data.
30. The method of claim 20 , wherein rearranging the elements in the one or more sets of data is performed by pruning the elements in the one or more sets of data.
31. A machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:
cause an increase in sparsity of one or more sets of data by at least rearranging elements in the one or more sets of data, such that the one or more sets of data comprise one or more subsets of data conforming to one or more structural sparsity constraints.
32. The machine-readable medium of claim 31 , wherein the set of instructions further comprise instructions that, if performed by the one or more processors, cause the one or more processors to rearrange the elements in the one or more sets of data by applying a set of transforms to the one or more sets of data according to the one or more structural sparsity constraints.
33. The machine-readable medium of claim 32 , wherein the one or more structural sparsity constraints comprise at least a limitation on the one or more sets of data, the limitation requiring the one or more subsets of data to comprise at least a first quantity of non-zero data values and a second quantity of zero data values.
34. The machine-readable medium of claim 32 , wherein the set of transforms comprise at least one transform exchanging a first subset of the one or more sets of data having a first position value in the one or more sets of data with a second subset of the one or more sets of data having a second position value in the one or more sets of data.
35. The machine-readable medium of claim 31 , wherein the set of instructions further comprise instructions that, if performed by the one or more processors, cause the one or more processors to determine a set of transforms to rearrange the elements in the one or more sets of data by:
randomly selecting a first subset of the one or more sets of data associated with a first position value and a second subset of the one or more sets of data associated with a second position value;
exchanging the first subset and the second subset;
setting one or more data values in the first subset and the second subset to a zero value;
calculating a metric value associated with a neural network corresponding to the one or more sets of data; and
as a result of the metric value being greater than another metric value, adding a transform comprising the first position value and the second position value to the set of transforms.
36. The machine-readable medium of claim 31 , wherein the set of instructions further comprise instructions that, if performed by the one or more processors, cause the one or more processors to rearrange the elements in the one or more sets of data by selecting a first subset of the one or more sets of data having a first position value and a second subset of the one or more sets of data having a second position and exchanging the first subset and second subset to increase a quantity of a class of data values in the one or more sets of data to be pruned in accordance with the one or more structural sparsity constraints.
37. The machine-readable medium of claim 31 , wherein the one or more processors further comprise one or more sparse tensor cores to perform one or more computations on the one or more sets of data based, at least in part, on the increase in sparsity of the one or more sets of data.
38. The machine-readable medium of claim 31 , wherein the set of instructions further comprise instructions that, if performed by the one or more processors, cause the one or more processors to cause the increase in sparsity by setting one or more data values in a subset of the one or more sets of data to a zero data value such that one or more operands of one or more computations on the one or more sets of data satisfy the one or more structural sparsity constraints.
39. The machine-readable medium of claim 31 , wherein the one or more sets of data are associated with one or more layers of a neural network having an improved accuracy as a result of the increase in sparsity.
40. The machine-readable medium of claim 31 , wherein rearranging the elements in the one or more sets of data is performed by permuting the elements in the one or more sets of data.
41. The machine-readable medium of claim 31 , wherein rearranging the elements in the one or more sets of data is performed by pruning the elements in the one or more sets of data.