Decomposing convolution operation in neural networks

1. A method of training an artificial neural network, comprising:

initializing the artificial neural network, the artificial neural network comprising at least one filter with a first rank, the first rank being a high rank;

adjusting the at least one filter by training the at least one filter in the artificial neural network to converge to a second rank, the second rank being a low rank that is less than the first rank, the training comprising:

dividing the at least one filter having a filter weight matrix into a linear combination of a row filter corresponding to a first neuron and column filter corresponding to a second neuron, the filter weight matrix comprising derived variables based on a free variable of the row filter and a free variable of the column filter; and

applying a gradient update to train the at least one filter to converge to the second rank, the gradient update determined with respect to matrix elements corresponding to the row filter and the column filter, in which updating the gradient comprises:

generating partial gradients with respect to the derived variables; and

generating partial gradients with respect to the free variable of the row filter and the free variable of the column filter using the partial gradients with respect to the derived variables; and

classifying an input to the artificial neural network based on a product generated by a convolution performed with the at least one trained filter, a number of convolution operations performed by the at least one trained filter with the second rank being less than a number of convolution operations performed by the at least one filter with the first rank.

2. The method of claim 1 , further comprising performing a decomposed convolution with the at least one trained filter to classify the input to the artificial neural network.

3. An apparatus for training an artificial neural network, comprising:

a memory; and

at least one processor coupled to the memory, the at least one processor configured:

to initialize the artificial neural network, the artificial neural network comprising at least one filter with a first rank, the first rank being a high rank;

to adjust the at least one filter by training the at least one filter in the artificial neural network to converge to a second rank, the second rank being a low rank that is less than the first rank, by:

dividing the at least one filter having a filter weight matrix into a linear combination of a row filter corresponding to a first neuron and column filter corresponding to a second neuron, the filter weight matrix comprising derived variables based on a free variable of the row filter and a free variable of the column filter; and

applying a gradient update to train the at least one filter to converge to the second rank, the gradient update determined with respect to matrix elements corresponding to the row filter and the column filter, in which updating the gradient comprises:

generating partial gradients with respect to the derived variables; and

generating partial gradients with respect to the free variable of the row filter and the free variable of the column filter using the partial gradients with respect to the derived variables; and

to classify an input to the artificial neural network based on a product generated by a convolution performed with the at least one trained filter, a number of convolution operations performed by the at least one trained filter with the second rank being less than a number of convolution operations performed by the at least one filter with the first rank.

4. The apparatus of claim 3 , in which the at least one processor is further configured to perform a decomposed convolution with the at least one trained filter to classify the input to the artificial neural network.

5. An apparatus for training an artificial neural network, comprising:

means for initializing the artificial neural network, the artificial neural network comprising at least one filter with a first rank, the first rank being a high rank;

means for adjusting the at least one filter by training the at least one filter in the artificial neural network to converge to a second rank, the second rank being a low rank that is less than the first rank, the means for training comprising:

means for dividing the at least one filter having a filter weight matrix into a linear combination of a row filter corresponding to a first neuron and column filter corresponding to a second neuron, the weight matrix comprising derived variables based on a free variable of the row filter and a free variable of the column filter; and

means for applying a gradient update to train the at least one filter to converge to the second rank, the gradient update determined with respect to matrix elements corresponding to the row filter and the column filter, in which updating the gradient comprises:

generating partial gradients with respect to the derived variables; and

generating partial gradients with respect to the free variable of the row filter and the free variable of the column filter using the partial gradients with respect to the derived variables; and

means for classifying an input to the artificial neural network based on a product generated by a convolution performed with the at least one trained filter, a number of convolution operations performed by the at least one trained filter with the second rank being less than a number of convolution operations performed by the at least one filter with the first rank.

6. The apparatus of claim 5 , further comprising means for performing a decomposed convolution with the at least one trained filter to classify the input to the artificial neural network.

7. A non-transitory computer readable medium having encoded thereon for training an artificial neural network, the program code executed by a processor and comprising:

program code to initialize the artificial neural network, the artificial neural network comprising at least one filter with a first rank, the first rank being a high rank;

program code to adjust the at least one filter by training the at least one filter in the artificial neural network to converge to a second rank, the second rank being a low rank that is less than the first rank, the program code to train comprising:

program code to divide the at least one filter having a filter weight matrix into a linear combination of a row filter corresponding to a first neuron and column filter corresponding to a second neuron, the filter weight matrix comprising derived variables based on a free variable of the row filter and a free variable of the column filter; and

program code to apply a gradient update to train the at least one filter to converge to the second rank, the gradient update determined with respect to matrix elements corresponding to the row filter and the column filter, in which updating the gradient comprises:

generating partial gradients with respect to the derived variables; and

generating partial gradients with respect to the free variable of the row filter and the free variable of the column filter using the partial gradients with respect to the derived variables; and

program code to classify an input to the artificial neural network based on a product generated by a convolution performed with the at least one trained filter, a number of convolution operations performed by the at least one trained filter with the second rank being less than a number of convolution operations performed by the at least one filter with the first rank.

8. The non-transitory computer readable medium of claim 7 , further comprising program code to perform a decomposed convolution with the at least one trained filter to classify the input to the artificial neural network.