Method and apparatus for universal pruning and compression of deep convolutional neural networks under joint sparsity constraints
A method and system for constructing a convolutional neural network (CNN) model are herein disclosed. The method includes regularizing spatial domain weights, providing quantization of the spatial domain weights, pruning small or zero weights in a spatial domain, fine-tuning a quantization codebook, compressing a quantization output from the quantization codebook, and decompressing the spatial domain weights and using either sparse spatial domain convolution and sparse Winograd convolution after pruning Winograd-domain weights.
1. A method of constructing a convolutional neural network (CNN) model to process an image, comprising:
regularizing spatial domain weights across at least one portion of the image;
providing quantization of the spatial domain weights;
pruning small weights in a spatial domain across the at least one portion of the image;
fine-tuning a quantization codebook based on the quantized spatial domain weights and the small weights in the spatial domain;
compressing a quantization output from the quantization codebook;
decompressing the quantization output using either sparse spatial domain convolution and sparse Winograd convolution after pruning Winograd-domain weights; and
processing the image using the CNN model,
wherein, if Winograd convolution is supported, the spatial domain weights are converted to the Winograd-domain weights, pruned to a desired sparsity level, and used to perform Winograd convolution.
2. The method of claim 1 , wherein the spatial domain weights are regularized using Winograd domain regularization.
3. The method of claim 2 , wherein regularizing the spatial domain weights further comprises using a learnable Winograd domain regularization coefficient and its penalty term.
4. The method of claim 2 , wherein regularizing the spatial domain weights further comprises defining a cost function with the Winograd domain regularizer and utilizing a gradient descent.
5. The method of claim 1 , wherein the spatial domain weights are regularized using spatial domain regularization.
6. The method of claim 5 , wherein regularizing the spatial domain weights further comprises using a learnable spatial domain regularization coefficient and its penalty term.
7. The method of claim 5 , wherein regularizing the spatial domain weights further comprises defining a cost function with the spatial domain regularizer and utilizing a gradient descent.
8. The method of claim 1 , wherein the quantization codebook is fine-tuned with Winograd domain regularization having non-zero code values and a quantized value is updated in the quantization codebook by gradient descent using an average gradient.
9. The method of claim 1 , wherein the CNN is operated in either the spatial domain or the Winograd domain after pruning.
10. The method of claim 1 , wherein decompressing further comprises transforming decompressed spatial domain filters to the Winograd domain and pruning the Winograd domain weights.
11. A system for constructing a convolutional neural network (CNN) model to process an image, comprising:
a processor; and
a non-transitory computer readable storage medium configured to store instructions that, when executed, cause the processor to:
regularize spatial domain weights across at least one portion of the image;
provide quantization of the spatial domain weights;
prune small weights in a spatial domain across the at least one portion of the image;
fine-tune a quantization codebook based on the quantized spatial domain weights and the small weights in the spatial domain;
compress a quantization output from the quantization codebook;
process the image using the CNN model; and
decompress the quantization output and use either sparse spatial domain convolution and sparse Winograd convolution after pruning Winograd-domain weights,
wherein, if Winograd convolution is supported, the spatial domain weights are converted to the Winograd-domain weights, pruned to a desired sparsity level, and used to perform Winograd convolution.
12. The system of claim 11 , wherein the spatial domain weights are regularized using Winograd domain regularization.
13. The system of claim 12 , wherein the instructions, when executed, further cause the processor to regularize the spatial domain weights by using a learnable Winograd domain regularization coefficient and its penalty term.
14. The system of claim 12 , wherein the instructions, when executed, further cause the processor to regularize the spatial domain weights by defining a cost function with the Winograd domain regularizer and using a gradient descent.
15. The system of claim 11 , wherein the spatial domain weights are regularized using spatial domain regularization.
16. The system of claim 15 , wherein the instructions, when executed, further cause the processor to regularize by using a learnable spatial domain regularization coefficient and its penalty term.
17. The system of claim 15 , wherein the instructions, when executed, further cause the processor to regularize the spatial domain weights by defining a cost function with the spatial domain regularizer and using a gradient descent.
18. The system of claim 11 , wherein the quantization codebook is fine-tuned with Winograd domain regularization and a quantized value is updated in the quantization codebook by gradient descent using an average gradient.
19. The system of claim 11 , wherein the CNN is operated in either the spatial domain or the Winograd domain after pruning.
20. The system of claim 11 , wherein the instructions, when executed, further cause the processor to decompress by transforming decompressed spatial domain filters to the Winograd domain and pruning the Winograd domain weights.