Efficient neural networks with elaborate matrix structures in machine learning environments
A mechanism is described for facilitating slimming of neural networks in machine learning environments. A method includes learning a first neural network associated with machine learning processes to be performed by a processor of a computing device, where learning includes analyzing a plurality of channels associated with one or more layers of the first neural network. The method may further include computing a plurality of scaling factors to be associated with the plurality of channels such that each channel is assigned a scaling factor, wherein each scaling factor to indicate relevance of a corresponding channel within the first neural network. The method may further include pruning the first neural network into a second neural network by removing one or more channels of the plurality of channels having low relevance as indicated by one or more scaling factors of the plurality of scaling factors assigned to the one or more channels.
1. An apparatus comprising:
one or more processors coupled to memory, the one or more processors configured to process instructions which, when executed, cause the one or more processors to:
facilitate selection of one or more paths associated with one or more matrix structures for a neural network associated with deep learning processes performed by the one or more processors; and
train the neural network based on the one or more paths and the one or more matrix structures to facilitate customization of the deep learning processes, wherein the one or more paths comprise one or more of a binary path based on a binary matrix structure or a ternary path based on a ternary matrix structure, wherein the deep learning processes are scaled based on one or more scaling factors associated with one or more channels such that one or more processing resources are dynamically enabled or disabled according to one or more of computational demands of a workload or processing capabilities and limitations of the apparatus, wherein the neural network and processing capabilities and limitations of the apparatus are observed to customize the one or more paths and the one or more matrix structures for the neural network, wherein the one or more matrix structures are specifically enforced on the neural network based on satisfaction of one or more conditions to reduce processing time and computation costs associated with convolutional layers of the neural network, wherein the one or more matrix structures include one or more of a symmetric matrix structure, a specific band matrix structure, a circulant matrix structure, or a Toeplitz matrix structure.
2. The apparatus of claim 1 , wherein the customization of the deep learning processes comprises scaling the deep learning processes according to the processing capabilities and limitations of the apparatus, wherein the one or more paths comprise one or more of a full circulant path based on a full circulant matrix structure, a Toeplitz path based on a Toeplitz matrix structure, a symmetric path based on a symmetric matrix structure, or a real weight path based on a real weight matrix structure.
3. The apparatus of claim 1 , wherein the one or more processors are further to simplify the neural network into a structure of one or more of an input convolutional layer, an output convolutional layer, and weights to prepare and allow for application of the one or more matrix structures to the neural network, and wherein the one or more paths comprise one or more of a full circulant path based on a full circulant matrix structure, a Toeplitz path based on a Toeplitz matrix structure, a symmetric path based on a symmetric matrix structure, or a real weight path based on a real weight matrix structure.
4. The apparatus of claim 1 , wherein the one or more processors comprises one or more graphics processors, wherein the one or more graphics processors are co-located with one or more application processors on a common semiconductor package.
5. A method comprising:
facilitating, by a processor of a computing device, selection of one or more paths associated with one or more matrix structures for a neural network associated with deep learning processes; and
training, by the processor, the neural network based on the one or more paths and the one or more matrix structures to facilitate customization of the deep learning processes, wherein the one or more paths comprise one or more of a binary path based on a binary matrix structure or a ternary path based on a ternary matrix structure, wherein the deep learning processes are scaled based on one or more scaling factors associated with one or more channels such that one or more processing resources are dynamically enabled or disabled according to one or more of computational demands of a workload or processing capabilities and limitations of the computing device, wherein the neural network and processing capabilities and limitations of the computing device are observed to customize the one or more paths and the one or more matrix structures for the neural network, wherein the one or more matrix structures are specifically enforced on the neural network based on satisfaction of one or more conditions to reduce processing time and computation costs associated with convolutional layers of the neural network, wherein the one or more matrix structures include one or more of a symmetric matrix structure, a specific band matrix structure, a circulant matrix structure, or a Toeplitz matrix structure.
6. The method of claim 5 , wherein the customization of the deep learning processes comprises scaling the deep learning processes according to the processing capabilities and limitations of the computing device.
7. The method of claim 5 , further comprising simplifying the neural network into a structure of one or more of an input convolutional layer, an output convolutional layer, and weights to prepare and allow for application of the one or more matrix structures to the neural network, wherein the one or more paths comprise one or more of a full circulant path based on a full circulant matrix structure, a Toeplitz path based on a Toeplitz matrix structure, a symmetric path based on a symmetric matrix structure, or a real weight path based on a real weight matrix structure.
8. The method of claim 5 , wherein the processor comprises a graphics processor, wherein the graphics processor is co-located with an application processor on a common semiconductor package.
9. At least one non-transitory machine-readable medium comprising instructions that when executed by a computing device, cause the computing device to perform operations comprising:
facilitating selection of one or more paths associated with one or more matrix structures for a neural network associated with deep learning processes performed by a processor of the computing device; and
training the neural network based on the one or more paths and the one or more matrix structures to facilitate customization of the deep learning processes, wherein the one or more paths comprise one or more of a binary path based on a binary matrix structure or a ternary path based on a ternary matrix structure, wherein the deep learning processes are scaled based on one or more scaling factors associated with one or more channels such that one or more processing resources are dynamically enabled or disabled according to one or more of computational demands of a workload or processing capabilities and limitations of the computing device, wherein the neural network and processing capabilities and limitations of the computing device are observed to customize the one or more paths and the one or more matrix structures for the neural network, wherein the one or more matrix structures are specifically enforced on the neural network based on satisfaction of one or more conditions to reduce processing time and computation costs associated with convolutional layers of the neural network, wherein the one or more matrix structures include one or more of a symmetric matrix structure, a specific band matrix structure, a circulant matrix structure, or a Toeplitz matrix structure.
10. The non-transitory machine-readable medium of claim 9 , wherein the customization of the deep learning processes comprises scaling the deep learning processes according to the processing capabilities and limitations of the computing device.
11. The non-transitory machine-readable medium of claim 9 , wherein the operations further comprise simplifying the neural network into a structure of one or more of an input convolutional layer, an output convolutional layer, and weights to prepare and allow for application of the one or more matrix structures to the neural network, wherein the one or more paths comprise one or more of a full circulant path based on a full circulant matrix structure, a Toeplitz path based on a Toeplitz matrix structure, a symmetric path based on a symmetric matrix structure, or a real weight path based on a real weight matrix structure.
12. The non-transitory machine-readable medium of claim 9 , wherein the processor comprises a graphics processor, wherein the graphics processor is co-located with an application processor on a common semiconductor package.