Computer-implemented methods and systems for compressing deep neural network models using alternating direction method of multipliers (ADMM)
ADMM-NN is an algorithm-hardware co-optimization framework of DNNs using Alternating Direction Method of Multipliers (ADMM). The first part of ADMM-NN is a systematic, joint framework of DNN weight pruning and quantization using ADMM. The second part is a hardware-aware optimization to facilitate hardware-level implementations. ADMM-based weight pruning and quantization accounts for (i) computation reduction and energy efficiency improvement and (ii) performance overhead due to irregular sparsity. Experimental results demonstrate that by combining weight pruning and quantization, the proposed framework can achieve 1,910× and 231× reductions in the overall model size on the LeNet-5 and AlexNet models. Favorable results are also observed on VGGNet and ResNet models. Also, without any accuracy loss, 3.6× reduction in the amount of computation can be achieved, outperforming prior work.
1 . A computer-implemented method for compressing a deep neural network (DNN) model using an Alternating Direction Method of Multipliers (ADMM) technique, comprising the steps of:
(a) performing ADMM weight pruning on weights in the DNN model by solving a first subproblem and a second subproblem iteratively until convergence to reduce a number of weights in the DNN model to a set of non-zero weights, wherein performing the ADMM weight pruning is based on a break-even pruning ratio, wherein the break-even pruning ratio is hardware platform specific, and calculating the break-even pruning ratio comprises,
synthesizing a baseline hardware implementation and at least a second hardware implementation, each having a same hardware area;
determining a first inference delay of the baseline hardware implementation of an original DNN layer without pruning and at least a second inference delay of at least the second hardware implementation under at least a second pruning ratio; and
determining, based on the first inference delay and at least the second inference delay, the break-even pruning ratio;
(b) performing ADMM weight quantization on the set of non-zero weights by solving the first subproblem and the second subproblem iteratively until convergence to generate a weight pruned and quantized DNN model;
wherein the first subproblem comprises performing DNN training with an additional L2 regularization term, and the second subproblem comprises dynamically updating a regularization target in each iteration; and
(c) allocating on-chip or off-chip memory to each portion of the weight pruned and quantized DNN model based on a computational intensity of that portion.
2 . The method of claim 1 , wherein the first subproblem comprises minimizing a loss function of the DNN model with the additional L2 regularization term.
3 . The method of claim 1 , wherein the first subproblem is solved using a stochastic gradient descent algorithm.
4 . The method of claim 1 , wherein the second subproblem is solved analytically and using optimization.
5 . The method of claim 1 , wherein the DNN model is pretrained.
6 . The method of claim 1 , further comprising performing a hardware-aware optimization of the weight pruned and quantized DNN model to facilitate an efficient hardware platform specific implementation.
7 . The method of claim 6 , wherein performing the hardware-aware optimization comprises performing further weight pruning and quantization of the DNN model to account for computation reduction and energy efficiency improvement and hardware performance overhead due to irregular sparsity.
8 . A computer system, comprising:
at least one processor;
memory associated with the at least one processor; and
a program supported in the memory for compressing a deep neural network (DNN) model using an Alternating Direction Method of Multipliers (ADMM) technique, the program containing a plurality of instructions which, when executed by the at least one processor, cause the at least one processor to:
(a) perform ADMM weight pruning on weights in the DNN model by solving a first subproblem and a second subproblem iteratively until convergence to reduce a number of weights in the DNN model to a set of non-zero weights, wherein performing the ADMM weight pruning is based on a break-even pruning ratio, wherein the break-even pruning ratio is hardware platform specific, and calculating the break-even pruning ratio comprises,
synthesizing a baseline hardware implementation and at least a second hardware implementation, each having a same hardware area;
determining a first inference delay of the baseline hardware implementation of an original DNN layer without pruning and at least a second inference delay of at least the second hardware implementation under at least a second pruning ratio; and
determining, based on the first inference delay and at least the second inference delay, the break-even pruning ratio;
(b) perform ADMM weight quantization on the set of non-zero weights by solving the first subproblem and the second subproblem iteratively until convergence to generate a weight pruned and quantized DNN model;
wherein the first subproblem comprises performing DNN training with an additional L2 regularization term, and the second subproblem comprises dynamically updating a regularization target in each iteration; and
(c) allocating on-chip or off-chip memory, in the memory associated with the at least one processor, to each portion of the weight pruned and quantized DNN model based on a computational intensity of that portion.
9 . The computer system of claim 8 , wherein the first subproblem comprises minimizing a loss function of the DNN model with the additional L2 regularization term.
10 . The computer system of claim 8 , wherein the first subproblem is solved using a stochastic gradient descent algorithm.
11 . The computer system of claim 8 , wherein the second subproblem is solved analytically and using optimization.
12 . The computer system of claim 8 , wherein the DNN model is pretrained.
13 . The computer system of claim 8 , further comprising performing a hardware-aware optimization of the weight pruned and quantized DNN model to facilitate an efficient hardware platform specific implementation.
14 . The computer system of claim 13 , wherein performing the hardware-aware optimization comprises performing further weight pruning and quantization of the DNN model to account for computation reduction and energy efficiency improvement and hardware performance overhead due to irregular sparsity.
15 . A computer program product for compressing a deep neural network (DNN) model using an Alternating Direction Method of Multipliers (ADMM) technique, the computer program product residing on a non-transitory computer readable medium having a plurality of instructions stored thereon which, when executed by a computer processor, cause that computer processor to:
(a) perform ADMM weight pruning on weights in the DNN model by solving a first subproblem and a second subproblem iteratively until convergence to reduce a number of weights in the DNN model to a set of non-zero weights, wherein performing the ADMM weight pruning is based on a break-even pruning ratio, wherein the break-even pruning ratio is hardware platform specific, and calculating the break-even pruning ratio comprises,
synthesizing a baseline hardware implementation and at least a second hardware implementation, each having a same hardware area;
determining a first inference delay of the baseline hardware implementation of an original DNN layer without pruning and at least a second inference delay of at least the second hardware implementation under at least a second pruning ratio; and
determining, based on the first inference delay and at least the second inference delay, the break-even pruning ratio;
(b) perform ADMM weight quantization on the set of non-zero weights by solving the first subproblem and the second subproblem iteratively until convergence to generate a weight pruned and quantized DNN model;
wherein the first subproblem comprises performing DNN training with an additional L2 regularization term, and the second subproblem comprises dynamically updating a regularization target in each iteration; and
(c) allocating on-chip or off-chip memory to each portion of the weight pruned and quantized DNN model based on a computational intensity of that portion.
16 . The computer program product of claim 15 , wherein the first subproblem comprises minimizing a loss function of the DNN model with the additional L2 regularization term.
17 . The computer program product of claim 15 , wherein the first subproblem is solved using a stochastic gradient descent algorithm.
18 . The computer program product of claim 15 , wherein the second subproblem is solved analytically and using optimization.
19 . The computer program product of claim 15 , further comprising instructions for performing a hardware-aware optimization of the weight pruned and quantized DNN model to facilitate an efficient hardware platform specific implementation.
20 . The computer program product of claim 19 , wherein performing the hardware-aware optimization comprises performing further weight pruning and quantization of the DNN model to account for computation reduction and energy efficiency improvement and hardware performance overhead due to irregular sparsity.