Techniques for increasing activation sparsity in artificial neural networks

1 . A method for implementing an artificial neural network in a computing system, comprising: performing a compute operation of a layer in the artificial neural network to generate output activations of the layer from input activations and weights of the layer, the artificial neural network comprising an activation function layer preceding the layer;

modifying the layer to increase activation sparsity, wherein modifying the layer comprises:

adding a first operation after the compute operation, and adding a second operation after the first operation, wherein the second operation is before the activation function layer;

performing the first operation by adding a bias value to the output activations to generate bias adjusted values; and

performing the second operation to generate modified output activations, wherein performing the second operation comprises adding one or more noise values to the bias adjusted values to modify a probability density function of an input of the activation function layer, the input of the activation function layer comprising the modified output activations,

wherein the modified output activations have increased sparsity than the output activations.

2 . The method of claim 1 , wherein the one or more noise values are generated by digital calculations.

3 . The method of claim 1 , wherein multiple input noise values are summed together to generate a noise value of the one or more noise values.

4 . The method of claim 1 further comprising:

generating a noise value of the one or more noise values using a linear feedback shift register.

5 . The method of claim 1 further comprising:

generating a Gaussian random variable for the one or more noise values using a Box-Muller transform.

6 . The method of claim 1 , wherein the method is implemented by an accelerator on at least one programmable logic integrated circuit.

7 . A non-transitory computer readable storage medium storing instructions executable to perform operations for implementing an artificial neural network, to the operations comprising:

performing a compute operation of a layer in the artificial neural network to generate output activations of the layer from input activations and weights of the layer, the artificial neural network comprising an activation function layer preceding the layer;

modifying the layer to increase activation sparsity, wherein modifying the layer comprises:

adding a first operation after the compute operation, and

adding a second operation after the first operation, wherein the second operation is before the activation function layer;

performing the first operation by adding a bias value to the output activations to generate bias adjusted values; and

performing the second operation to generate modified output activations, wherein performing the second operation comprises adding one or more noise values to the bias adjusted values to modify a probability density function of an input of the activation function layer, the input of the activation function layer comprising the modified output activations,

wherein the modified output activations have increased sparsity than the output activations.

8 . The non-transitory computer readable storage medium of claim 7 , wherein the activation function layer has a rectified linear activation function.

9 . The non-transitory computer readable storage medium of claim 7 , wherein the operations further comprise:

generating the one or more noise values using a piecewise linear function.

10 . The non-transitory computer readable storage medium of claim 7 , wherein the operations further comprise:

generating the one or more noise values using at least one linear feedback shift register.

11 . The non-transitory computer readable storage medium of claim 7 , wherein the operations further comprise:

generating the one or more noise values with a central limit theorem that generates a cumulative distribution function that approximates a Gaussian distribution.

12 . The non-transitory computer readable storage medium of claim 7 , wherein the operations further comprise:

generating a Gaussian random variable for the one or more noise values using a Box-Muller transform.

13 . A computing system for implementing an artificial neural network, wherein the computing system comprises: a computer processor for executing computer program instructions; and a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to:

perform a compute operation of a layer in the artificial neural network to generate output activations of the layer from input activations and weights of the layer, the artificial neural network comprising an activation function layer preceding the layer,

modifying the layer to increase activation sparsity, wherein modifying the layer comprises:

adding a first operation after the compute operation, and

adding a second operation after the first operation, wherein the second operation is before the activation function layer,

performing the first operation by adding a bias value to the output activations to generate bias adjusted values, and

performing the second operation to generate modified output activations, wherein performing the second operation comprises adding one or more noise values to the bias adjusted values to modify a probability density function of an input of the activation function layer, the input of the activation function layer comprising the modified output activations,

wherein the modified output activations have increased activation sparsity than the output activations.

14 . The computing system of claim 13 , wherein the activation function layer has a rectified linear activation function.

15 . The method of claim 1 , wherein the activation function layer has a rectified linear unit operation, wherein the method further comprises performing the rectified linear unit operation on the modified output activations.

16 . The method of claim 1 , wherein the compute operation is a convolution operation.

17 . The method of claim 1 , wherein the input activations are in an input tensor of the layer, wherein the one or more noise values are in a noise tensor, the noise tensor having a same dimension as the input tensor.

18 . The computing system of claim 13 , wherein the compute operation is a convolution operation.

19 . The computing system of claim 13 , wherein the input activations are in an input tensor of the layer, wherein the one or more noise values are in a noise tensor, the noise tensor having a same dimension as the input tensor.

20 . The computing system of claim 13 , wherein the one or more noise values are generated using at least one linear feedback shift register.