Variance aware techniques to prune neural networks

1 . One or more processors, comprising:

circuitry to:

identify one or more filters of one or more first layers of a neural network that are to be removed from each of the one or more first layers;

perform a regularization on the one or more filters;

group a set of filters of the one or more filters based on alignment of the set of filters across two or more layers of the neural network;

align structure sparsity across the two or more layers based, at least in part, on the regularization on the one or more filters;

determine, as a result of the regularization, whether the set of filters satisfy a pruning criteria; and

deactivate one or more second layers of the neural network in response to determining the set of filters satisfies the pruning criteria by pruning the neural network to remove the set of filters from the neural network.

2 . The one or more processors of claim 1 , wherein the circuitry is to remove the one or more filters by at least:

applying the sparsity across the set of filters based, at least in part, on the regularization on the one or more filters; and

determining that, as a result of the regularization, the set of filters is sufficiently sparse to be removed.

3 . The one or more processors of claim 2 , wherein performing the regularization to enforce aligned structure sparsity across the two or more layers comprises penalizing variance of weights among the two or more layers.

4 . The one or more processors of claim 1 , wherein one or more second layers comprise two or more layers connected by skip-connections.

5 . The one or more processors of claim 1 , wherein the neural network comprising the one or more first layers and one or more second layers is a residual neural network.

6 . The one or more processors of claim 1 , wherein a first layer of the one or more first layers and a second layer of the one or more first layers are directly connected by an element-wise operator.

7 . The one or more processors of claim 1 , wherein, to perform inferencing using the neural network, the one or more second layers depend on the one or more first layers.

8 . The one or more processors of claim 1 , wherein, to perform inferencing using the neural network, the one or more first layers depends on the one or more second layers.

9 . The one or more processors of claim 1 , wherein deactivation of a set of filters comprises an effect on accuracy comparison generated based, at least in part, on the one or more first layers.

10 . The one or more processors of claim 1 , wherein the pruning criteria is associated with a threshold value.

11 . The one or more processors of claim 1 , wherein the circuitry is to compare one or more values of the one or more filters to a threshold.

12 . The one or more processors of claim 1 , wherein weights of the one or more first layers comprises one or more first filters that are modified to compensate removal of one or more second filters of the one or more second layers.

13 . The one or more processors of claim 1 , wherein deactivation of the set of filters comprises identifying a first accuracy after one or more first filters are deactivated, identifying a second accuracy after one or more second filters are deactivated, and determining which of the one or more first filters and the one or more second filters is to be removed based, at least in part, on the identification.

14 . A system, comprising:

one or more computers comprising one or more processors to:

identify one or more filters of one or more first layers of a neural network that are to be removed from each of the one or more first layers;

perform a regularization on the one or more filters;

group a set of filters of the one or more filters based on alignment of the set of filters across two or more layers of the neural network;

align structure sparsity across the two or more layers based, at least in part, on the regularization on the one or more filters;

determine, as a result of the regularization, whether the set of filters satisfy a pruning criteria; and

deactivate one or more second layers of the neural network in response to determining the set of filters satisfies the pruning criteria by pruning the neural network to remove the set of filters from the neural network.

15 . The system of claim 14 , wherein the one or more processors remove the one or more filters by at least:

applying the sparsity across the set of filters based, at least in part, on the regularization on the one or more filters; and

determining that, as a result of applying the regularization, that the set of filters is sufficiently sparse to be removed.

16 . The system of claim 15 , wherein applying the regularization to the set of filters comprises, for a filter of the set of filters:

calculating a norm for a weight group of the set of filters; and

calculating a difference in weight values of the weight group.

17 . The system of claim 16 , wherein the determining that a first filter of the set of filters is to be removed with a second filter of the set of filters is based on a common index placement of the first filter and the second filter.

18 . The system of claim 15 , wherein the set of filters are from the one or more first layers that are connected by element-wise operators.

19 . The system of claim 15 , wherein determining that the set of filters is sufficiently sparse comprises comparing at least one norm of each filter of the set of filters with a threshold value.

20 . The system of claim 15 , wherein the one or more processors are further to prune the one or more second layers as a result of determining that the set of filters is sufficiently sparse to be removed.

21 . A non-transitory machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:

identify one or more filters of one or more first layers of a neural network that are to be removed from each of the one or more first layers;

perform a regularization on the one or more filters;

group a set of filters of the one or more filters based on alignment of the set of filters across two or more layers of the neural network;

align structure sparsity across the two or more layers based, at least in part, on the regularization on the one or more filters;

determine, as a result of the regularization, whether the set of filters satisfy a pruning criteria; and

deactivate one or more second layers of the neural network in response to determining the set of filters satisfies the pruning criteria by pruning the neural network to remove the set of filters from the neural network.

22 . The non-transitory machine-readable medium of claim 21 , wherein deactivation of a set of filters of the one or more second layers comprises removing the one or more second layers.

23 . The non-transitory machine-readable medium of claim 21 , wherein the one or more second layers are removed in response to the one or more second layers satisfying a pruning criteria.

24 . The non-transitory machine-readable medium of claim 21 , wherein the set of instructions, which if performed by the one or more processors, cause the one or more processors to adjust the one or more first layers based, at least in part, on aligning sparsity between the one or more second layers.

25 . The non-transitory machine-readable medium of claim 21 , wherein the neural network comprising the one or more first layers and one or more second layers comprise a ResNet block.

26 . The non-transitory machine-readable medium of claim 21 , wherein the one or more second layers are connected layers.

27 . The non-transitory machine-readable medium of claim 21 , wherein an accuracy of the neural network comprising the one or more first layers and one or more second layers is substantially unaffected by deactivation of the one or more second layers.

28 . One or more processors, comprising:

circuitry to help train a neural network by at least:

identifying one or more filters of one or more first layers of the neural network that are to be removed from each of the one or more first layers;

performing a regularization on the one or more filters;

grouping a set of filters of the one or more filters based on alignment of the set of filters across two or more layers of the neural network;

aligning structure sparsity across the two or more layers based, at least in part, on the regularization on the one or more filters;

determining, as a result of the regularization, whether the set of filters satisfy a pruning criteria; and

deactivating one or more second layers of the neural network in response to determining the set of filters satisfies the pruning criteria by pruning the neural network to remove the set of filters from the neural network.

29 . The one or more processors of claim 28 , wherein the circuitry is to adjust one or more weights of the neural network in order to reduce variation of weights in the one or more second layers of the neural network.

30 . The one or more processors of claim 29 , wherein adjusting the one or more weights of the neural network in order to reduce the variation of weights in the one or more second layers of the neural network comprises reducing at least one norm of the weights.

31 . The one or more processors of claim 28 , wherein the circuitry is to:

at a first stage of training, apply a 1-norm regularization on the neural network;

at a second stage of training, apply a regularization on a group of filters from a set of layers connected by element-wise operations; and

as a result of performing the first stage and the second stage, pruning the neural network.

32 . The one or more processors of claim 28 , wherein the neural network is to be trained to detect objects in images.

33 . The one or more processors of claim 28 , wherein the neural network is a residual neural network.

34 . A system, comprising:

one or more computers comprising one or more processors to help train a neural network by at least:

identifying one or more filters of one or more first layers of the neural network that are to be removed from each of the one or more first layers;

performing a regularization on the one or more filters;

grouping a set of filters of the one or more filters based on alignment of the set of filters across two or more layers of the neural network;

aligning structure sparsity across the two or more layers based, at least in part, on the regularization on the one or more filters;

determining, as a result of the regularization, whether the set of filters satisfy a pruning criteria; and

deactivating one or more second layers of the neural network in response to determining the set of filters satisfies the pruning criteria by pruning the neural network to remove the set of filters from the neural network.

35 . The system of claim 34 , wherein the one or more processors are to train the neural network by at least:

at a first stage of training, apply a regularization on at least one set of a set of the one or more first layers or a set of one or more second layers connected by skip-connections;

at a second stage of training, applying a 1-norm regularization on layers of a different set of layers of the neural network from the at least one set of layers connected by skip-connections; and

as a result of execution of a first layer and a second layer, pruning the neural network.

36 . The system of claim 35 , wherein the one or more processors to prune the neural network cause the neural network to remove two or more filters of the at least one set of layers connected by skip-connections.

37 . The system of claim 36 , wherein the two or more filters of the at least one of a set of the one or more first layers or a set of one or more second layers comprises a first filter and a second filter of the at least one of a set of the one or more first layers or a set of the one or more second layers at an index position.

38 . The system of claim 35 , wherein pruning the neural network generates a pruned neural network and the one or more processors are to further:

determine a baseline error rate of the neural network;

determine a pruned error rate of the pruned neural network; and

determine whether a threshold error is exceeded based on the baseline error rate and the pruned error rate.

39 . The system of claim 34 , wherein the one or more processors to use weight comparison corresponding to the neural network are to at least:

apply the sparsity across the set of filters based, at least in part, on the regularization on the one or more filters; and

determine that, as a result of the regularization, that the set of filters is sufficiently sparse to be removed.

40 . The system of claim 34 , wherein the system is an image recognition system, and the neural network is configured to obtain one or more images and, as a result of execution of the neural network, determine one or more classifications for the one or more images.

41 . A non-transitory machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least: help train a neural network by:

identifying one or more filters of one or more first layers of the neural network that are to be removed from each of the one or more first layers;

performing a regularization on the one or more filters;

grouping a set of filters of the one or more filters based on alignment of the set of filters across two or more layers of the neural network;

aligning structure sparsity across the two or more layers based, at least in part, on the regularization on the one or more filters;

determining, as a result of the regularization, whether the set of filters satisfy a pruning criteria; and

deactivating one or more second layers of the neural network in response to determining the set of filters satisfies the pruning criteria by pruning the neural network to remove the set of filters from the neural network.

42 . The non-transitory machine-readable medium of claim 41 , wherein the set of instructions to train the neural network, if performed by the one or more processors, cause the one or more processors to perform deactivation of the set of filters of the one or more second layers by at least pruning the neural network.

43 . The non-transitory machine-readable medium of claim 41 , wherein the set of instructions, which if performed by the one or more processors, cause the one or more processors to perform deactivation of the set of filters of the one or more second layers based at least in part on aligning sparsity between the one or more second layers.

44 . The non-transitory machine-readable medium of claim 41 , wherein the neural network comprises a ResNet block.

45 . The non-transitory machine-readable medium of claim 41 , wherein one or more second layers are layers directly connected by skip-connections.

46 . The non-transitory machine-readable medium of claim 41 , wherein an accuracy of the neural network is substantially unaffected by deactivation of the one or more second layers.