Pruning neural networks that include element-wise operations
Input layers of an element-wise operation in a neural network can be pruned such that the shape (e.g., the height, the width, and the depth) of the pruned layers matches. A pruning engine identifies all of the input layers into the element-wise operation. For each set of corresponding neurons in the input layers, the pruning engine equalizes the metrics associated with the neurons to generate an equalized metric associated with the set. The pruning engine prunes the input layers based on the equalized metrics generated for each unique set of corresponding neurons.
1. A computer-implemented method comprising:
computing an average weight value of a plurality of neurons of one or more neural networks, wherein each of the plurality of neurons is located within at a matching location within a different layer included in a plurality of locations of different layers of the one or more neural networks, and wherein each of the plurality of neurons is associated with a matching feature type;
replacing a weight value of each of the plurality of neurons with the average weight value; and
deactivating each of the plurality of neurons based, at least in part, on whether the replaced weight value of the plurality of neurons exceeds a threshold value.
2. The method of claim 1 , wherein one or more outputs of the plurality of different layers are processed using one or more element-wise operations.
3. The method of claim 1 , wherein deactivating each of the plurality of neurons comprises performing one or more equalization operations on weights associated with the plurality of neurons.
4. The method of claim 3 , wherein performing the one or more equalization operations comprises applying an equalization operator to the weights associated with the plurality of neurons.
5. The method of claim 1 , wherein the average weight value is computed using an arithmetic mean or a geometric mean of weights associated with the plurality of neurons.
6. The method of claim 1 , wherein deactivating each of a plurality of neurons further comprises:
obtaining a desired dimensionality of the plurality of different layers to be deactivated.
7. The method of claim 1 , wherein the one or more neural networks comprise a residual network, and wherein the plurality of different layers includes a convolutional layer of the residual network and an identity layer of the residual network.
8. The method of claim 1 , wherein each of the plurality of neurons produces a different input into a given computational component of the one or more neural networks.
9. The method of claim 1 , wherein the weight value is computed using an arithmetic mean or a geometric mean of the weights associated with the plurality of neurons.
10. One or more processors comprising circuitry to:
computing compute an average weight value of a plurality of neurons of one or more neural networks, wherein each of the plurality of neurons is located within at a matching location within a different layer included in a plurality of locations of different layers of one or more neural networks, and wherein each of the plurality of neurons is associated with a matching feature type;
replacing replace a weight value of each of the plurality of neurons with the average weight value; and
deactivate each of the plurality of neurons based, at least in part, on whether the replaced weight value of the plurality of neurons exceeds a threshold value.
11. The one or more processors of claim 10 , wherein one or more outputs of the plurality of different layers are processed using one or more element-wise operations.
12. The one or more processors of claim 10 , wherein deactivation of each of the plurality of neurons comprise computing L2 norm of weights associated with the plurality of neurons.
13. The one or more processors of claim 10 , wherein deactivation of each of the plurality of neurons comprises performing one or more equalization operations on weights associated with the plurality of neurons.
14. The one or more processors of claim 13 , wherein performing the one or more equalization operations comprises applying an equalization operator to the weights associated with the plurality of neurons.
15. The one or more processors of claim 10 , wherein the average weight value is computed using an arithmetic mean or a geometric mean of the weights associated with the plurality of neurons.
16. The one or more processors of claim 10 , wherein deactivation each of the plurality of neurons comprises:
obtaining a desired dimensionality of the plurality of different layers to be deactivated.
17. The one or more processors of claim 10 , wherein the one or more neural networks comprise a residual network, and wherein the plurality of different layers include a convolutional layer of the residual network and an identity layer of the residual network.
18. A system comprising: one or more processors to:
compute an average weight value of a plurality of neurons of one or more neural networks, wherein each of the plurality of neurons is located at a within matching location within a different layer included in a plurality of locations of different layers of the one or more neural networks, and wherein each of the plurality of neurons is associated with a matching feature type;
replace a weight value of each of the plurality of neurons with the average weight value; and
deactivate each of the plurality of neurons based, at least in part, on whether the replaced weight value of the plurality of neurons exceeds a threshold value.
19. The system of claim 18 , wherein one or more outputs of the plurality of different layers are processed using one or more element-wise operations.
20. The system of claim 18 , wherein the deactivation of the plurality of neurons comprises performing an equalization operation on weights associated with the plurality of neurons.
21. The system of claim 18 , wherein the one or more neural networks comprise a residual network, and wherein the plurality of different layers includes a convolutional layer of the residual network and an identity layer of the residual network.