Cluster compression for compressing weights in neural networks

1. A method, comprising:

instantiating a convolutional neural network on a computing system, the convolutional neural network including a plurality of layers, wherein instantiating the convolutional neural network comprises:

training the convolutional neural network using a first loss function until a first classification accuracy is reached, wherein the first loss function calculates a classification error of the convolutional neural network, wherein training the convolutional neural network with the first loss function comprises optimizing, for a first one of the layers, a first set of F filters and a first set of F biases so as to minimize the first loss function, wherein each of the F filters is formed from K kernels, wherein K and F are each greater than one, wherein each of the kernels consists of nine parameters and wherein each of the biases is scalar;

clustering the set of F×K kernels of the first layer into a set of C clusters, wherein each of the clusters is characterized by a centroid, thereby the C clusters being characterized by C centroids, wherein each of the centroids consists of nine parameters, and wherein C is less than F×K;

training the convolutional neural network using a second loss function until a second classification accuracy is reached;

creating a dictionary which maps each of the centroids to a corresponding scalar centroid identifier;

quantizing and compressing the F filters of the first layer by, for each of the F×K kernels, replacing the nine parameters of the kernel with one of the scalar centroid identifiers from the dictionary;

storing the F quantized and compressed filters of the first layer in a memory of the computing system, the F quantized and compressed filters comprising F×K scalar centroid identifiers; and

storing the F biases of the first layer in the memory; and

classifying data received by the convolutional neural network, wherein the classification comprises:

retrieving the F quantized and compressed filters of the first layer from the memory, the F quantized and compressed filters comprising the F×K scalar centroid identifiers;

decompressing, using the dictionary, the F quantized and compressed filters of the first layer into F quantized filters by mapping the F×K scalar centroid identifiers into F×K corresponding quantized kernels, the F×K corresponding quantized kernels each having nine parameters and forming the F quantized filters;

retrieving the F biases of the first layer from the memory; and

for the first layer, computing a convolution of the received data or data output from a layer previous to the first layer with the F quantized filters and the F biases, wherein a number of channels of the received data or data output from the layer previous to the first layer is equal to K.

2. The method of claim 1 , further comprising:

during the instantiation of the convolutional neural network, further storing the dictionary in the memory; and

during the classification of the received data, further retrieving the dictionary from the memory.

3. The method of claim 1 , wherein C is equal to 2 n , where n is a natural number.

4. The method of claim 3 , wherein each of the scalar centroid identifiers is expressed using n-bits.

5. The method of claim 1 , wherein the F×K kernels are clustered into the C clusters using the k-means algorithm.

6. The method of claim 1 , wherein each of the F×K kernels is represented by a 3×3 matrix of parameters.