Insertion of activation functions into decomposed layer of network

1 . A method comprising:

receiving a machine-trained (MT) network comprising a plurality of layers;

replacing a first layer of the plurality of layers with a second layer followed by a third layer, the first layer of the plurality of layers being associated with a set of activation functions, wherein replacing the first layer comprises:

decomposing a first plurality of weight values associated with the first layer into (i) a second plurality of weight values for the second layer and (ii) a third plurality of weight values for the third layer, wherein the third layer comprises the set of activation functions, and

inserting at least one additional activation function for the second layer into the MT network, wherein the at least one additional activation function is different than the set of activation functions,

providing input values to the MT network; and

generating, using the input values and based on executing the second layer including the second plurality of weight values and the third layer including the third plurality of weight values, an output for the MT network.

2 . The method of claim 1 ,

wherein inserting the at least one additional activation function increases expressiveness of the MT network.

3 . The method of claim 1 , wherein the first layer is decomposed to configure the MT network for execution by a particular type of neural network inference circuit.

4 . The method of claim 3 , wherein insertion of the at least one additional activation function does not modify an amount of time required for the particular type of neural network inference circuit to execute the MT network.

5 . The method of claim 4 , wherein the particular type of neural network inference circuit comprises:

a first set of sub-circuits for computing intermediate values based on input values and weight values; and

a second set of sub-circuits for applying activation functions to computed intermediate values,

wherein signals representing intermediate values are computed by the first set of sub-circuits pass through the second set of sub-circuits irrespective of whether an activation function is applied.

6 . The method of claim 1 , wherein:

the first plurality of weight values for the first layer comprises a first number of filters;

the second plurality of weight values for the second layer comprises a second number of filters that is less than the first number of filters; and

the third plurality of weight values for the third layer comprises the first number of filters.

7 . The method of claim 6 , wherein:

each filter of the second layer has a same number of weight values as each filter of the first layer;

each filter of the third layer comprises fewer weight values than of the first layer and each filter of the second layer; and

a total number of combined weight values in the second plurality of weight values and the third plurality of weight values is less than a total number of weight values in the first plurality of weight values.

8 . The method of claim 6 , wherein the third layer is a 1×1 convolutional layer, such that a number of weight values in each filter of the third layer is equal to the second number of filters in the second layer.

9 . The method of claim 6 , wherein:

each filter in the first layer has an associated stride and zero-padding;

each filter in the second layer has a same associated stride and zero-padding; and

each filter in the third layer is a 1×1 convolutional filter with a stride of 1 and no zero-padding.

10 . The method of claim 1 , wherein:

the first plurality of weight values are trained as floating-point weight values; and

each weight value of the second plurality of weight values are restricted to a ternary set of allowed weight values.

11 . The method of claim 1 ,

wherein decomposing the first plurality of weight values comprises performing singular value decomposition.

12 . The method of claim 1 further comprising training the second plurality of weight values and the third plurality of weight values.

13 . A non-transitory machine-readable medium storing a program which when executed by at least one processing unit trains a plurality of parameters of a network, the program comprising sets of instructions for:

receiving a machine-trained (MT) network comprising a plurality of layers;

replacing a first layer of the plurality of layers with a second layer followed by a third layer, the first layer of the plurality of layers being associated with a set of activation functions, wherein replacing the first layer comprises:

decomposing a first plurality of weight values associated with the first layer into (i) a second plurality of weight values for the second layer and (ii) a third plurality of weight values for the third layer, wherein the third layer comprises the set of activation functions, and

inserting at least one additional activation function for the second layer into the MT network, wherein the at least one additional activation function is different than the set of activation functions;

providing input values to the MT network; and

generating, using the input values and based on executing the second layer including the second plurality of weight values and the third layer including the third plurality of weight values, an output for the MT network.

14 . The non-transitory machine-readable medium of claim 13 ,

wherein insertion of the at least one additional activation function increases expressiveness of the MT network.

15 . The non-transitory machine-readable medium of claim 13 , wherein:

the MT network is for execution by a particular type of neural network inference circuit; and

insertion of the at least one additional activation function does not modify an amount of time required for the neural network inference circuit to execute the MT network.

16 . The non-transitory machine-readable medium of claim 15 , wherein the neural network inference circuit comprises:

a first set of sub-circuits for computing intermediate values based on input values and weight values; and

a second set of sub-circuits for applying activation functions to computed intermediate values,

wherein signals representing the intermediate values are computed by the first set of sub-circuits pass through the second set of sub-circuits irrespective of whether an activation function is applied.

17 . The non-transitory machine-readable medium of claim 13 , wherein:

the first plurality of weight values for the first layer comprises a first number of filters;

the second plurality of weight values for the second layer comprises a second number of filters that is less than the first number of filters; and

the third plurality of weight values for the third layer comprises the first number of filters.

18 . The non-transitory machine-readable medium of claim 17 , wherein:

each filter of the second layer has a same number of weight values as each filter of the first layer;

each filter of the third layer comprises fewer weight values than of the first layer and each filter of the second layer; and

a total number of combined weight values in the second plurality of weight values and the third plurality of weight values is less than a total number of weight values in the first plurality of weight values.

19 . The non-transitory machine-readable medium of claim 17 , wherein the third layer is a 1×1 convolutional layer, such that a number of weight values in each filter of the third layer is equal to the second number of filters in the second layer.

20 . The non-transitory machine-readable medium of claim 13 , wherein:

the first plurality of weight values are trained as floating-point weight values; and

each weight value of the second plurality of weight values are restricted to a ternary set of allowed weight values.