Variational dropout with smoothness regularization for neural network model compression
A method, computer program, and computer system is provided for compressing a deep neural network model. Weight coefficients associated with a deep neural network are quantize and entropy-coded. The quantized and entropy-coded weight coefficients are locally smoothed. The smoothed weight coefficients are compressed based on applying a variational dropout to the weight coefficients.
1. A method of compressing a deep neural network model, executable by a processor, the method comprising:
quantizing and entropy-coding weight coefficients associated with the deep neural network;
locally smoothing the quantized and entropy-coded weight coefficients; and
compressing the smoothed weight coefficients based on applying a variational dropout to the weight coefficients.
2. The method of claim 1 , wherein the weight coefficients correspond to dimensions associated with a multi-dimensional tensor.
3. The method of claim 1 , wherein the weight coefficients are locally smoothed based on determining a gradient associated with the weight coefficients.
4. The method of claim 1 , wherein the weight coefficients are locally smoothed based on minimizing a loss value associated with a smoothness metric of the weight coefficients.
5. The method of claim 4 , wherein the loss value is minimized based on selecting a subset of weight coefficients from among the weight coefficients.
6. The method of claim 4 , wherein the smoothness metric corresponds to an absolute value of gradients of the weight coefficients at one or more axes measured at a point based on balancing contributions of the gradients at the one or more axes.
7. The method of claim 4 , further comprising training the deep neural network based on the minimized loss value.
8. A computer system for compressing a deep neural network model, the computer system comprising:
one or more computer-readable non-transitory storage media configured to store computer program code; and
one or more computer processors configured to access said computer program code and operate as instructed by said computer program code, said computer program code including:
quantizing and entropy-coding code configured to cause the one or more computer processors to quantize and entropy-code weight coefficients associated with the deep neural network;
smoothing code configured to cause the one or more computer processors to locally smooth the quantized and entropy-coded weight coefficients; and
compressing code configured to cause the one or more computer processors to compress the smoothed weight coefficients based on applying a variational dropout to the weight coefficients.
9. The computer system of claim 8 , wherein the weight coefficients correspond to dimensions associated with a multi-dimensional tensor.
10. The computer system of claim 8 , wherein the weight coefficients are locally smoothed based on determining a gradient associated with the weight coefficients.
11. The computer system of claim 8 , wherein the weight coefficients are locally smoothed based on minimizing a loss value associated with a smoothness metric of the weight coefficients.
12. The computer system of claim 11 , wherein the loss value is minimized based on selecting a subset of weight coefficients from among the weight coefficients.
13. The computer system of claim 11 , wherein the smoothness metric corresponds to an absolute value of gradients of the weight coefficients at one or more axes measured at a point based on balancing contributions of the gradients at the one or more axes.
14. The computer system of claim 11 , further comprising training code configured to cause the one or more computer processors to train the deep neural network based on the minimized loss value.
15. A non-transitory computer readable medium having stored thereon a computer program for compressing a deep neural network model, the computer program configured to cause one or more computer processors to:
quantize and entropy-code weight coefficients associated with the deep neural network;
locally smooth the quantized and entropy-coded weight coefficients; and
compress the smoothed weight coefficients based on applying a variational dropout to the weight coefficients.
16. The computer readable medium of claim 15 , wherein the weight coefficients correspond to dimensions associated with a multi-dimensional tensor.
17. The computer readable medium of claim 15 , wherein the weight coefficients are locally smoothed based on determining a gradient associated with the weight coefficients.
18. The computer readable medium of claim 15 , wherein the weight coefficients are locally smoothed based on minimizing a loss value associated with a smoothness metric of the weight coefficients.
19. The computer readable medium of claim 18 , wherein the loss value is minimized based on selecting a subset of weight coefficients from among the weight coefficients.
20. The computer readable medium of claim 18 , wherein the smoothness metric corresponds to an absolute value of gradients of the weight coefficients at one or more axes measured at a point based on balancing contributions of the gradients at the one or more axes.