Jointly pruning and quantizing deep neural networks
A system and a method generate a neural network that includes at least one layer having weights and output feature maps that have been jointly pruned and quantized. The weights of the layer are pruned using an analytic threshold function. Each weight remaining after pruning is quantized based on a weighted average of a quantization and dequantization of the weight for all quantization levels to form quantized weights for the layer. Output feature maps of the layer are generated based on the quantized weights of the layer. Each output feature map of the layer is quantized based on a weighted average of a quantization and dequantization of the output feature map for all quantization levels. Parameters of the analytic threshold function, the weighted average of all quantization levels of the weights and the weighted average of each output feature map of the layer are updated using a cost function.
1 . A neural network, comprising a plurality of layers, at least one layer comprising jointly pruned and quantized weights and output feature maps, the jointly pruned weights being pruned using an analytic threshold function, each weight remaining after being pruned further being quantized based on a weighted average of a quantization and dequantization of the weight for all quantization levels.
2 . The neural network of claim 1 , wherein the output feature maps are formed based on pruned and quantized weights of the layer, each output feature map being quantized based on a weighted average of a quantization and dequantization of the output feature map for all quantization levels and parameters of the analytic threshold function, and
wherein the weighted average of all quantization levels of the weights and the weighted average of each output feature map of the layer are updated iteratively based on a cost function.
3 . The neural network of claim 2 , wherein the neural network is a full-precision trained neural network before weights and output feature maps of the at least one layer are jointly pruned and quantized.
4 . The neural network of claim 2 , wherein the cost function includes a pruning loss term, a weight quantization loss term and a feature map quantization loss term.
5 . The neural network of claim 2 , wherein the parameters of the analytic threshold function, the weighted average of all quantization levels of the weights and the weighted average of each output feature map of the layer are updated based on an optimization of the cost function.
6 . The neural network of claim 2 , wherein the parameters of the analytic threshold function, the weighted average of all quantization levels of the weights and the weighted average of each output feature map of the layer are iteratively updated based on an optimization of the cost function.
7 . The neural network of claim 2 , wherein the parameters of the analytic threshold function include a first parameter that controls a sharpness of the analytic threshold function, and a second parameter that controls a distance between a first edge and a second edge of the analytic threshold function.
8 . A method to prune weights and output feature maps of a layer of a neural network, the method comprising:
pruning weights of a layer of a neural network using an analytic threshold function, the neural network being a trained neural network;
quantizing each weight of the layer remaining after pruning based on a weighted average of a quantization and dequantization of the weight for all quantization levels to form quantized weights for the layer; and
determining output feature maps of the layer based on quantized weights of the layer.
9 . The method of claim 8 , further comprising:
quantizing each output feature map of the layer based on a weighted average of a quantization and dequantization of the output feature map for all quantization levels; and
iteratively updating parameters of the analytic threshold function, the weighted average of all quantization levels of the weights and the weighted average of each output feature map of the layer using a cost function.
10 . The method of claim 9 , wherein updating the parameters of the analytic threshold function, the weighted average of all quantization levels of the weights and the weighted average of each output feature map of the layer further comprises optimizing the cost function.
11 . The method of claim 10 , wherein the cost function includes a pruning loss term, a weight quantization loss term and a feature map quantization loss term.
12 . The method of claim 10 , further comprising iteratively pruning the weights, quantizing each weight of the layer, determining the output feature maps of the layer, quantizing each output feature map of the layer, and updating the parameters of the analytic threshold function, the weighted average of all quantization levels of the weights and the weighted average of each output feature map of the layer to optimize the cost function.
13 . The method of claim 12 , wherein the layer of the neural network is a first layer,
the method further comprising:
pruning weights of a second layer of the neural network using the analytic threshold function, the second layer being subsequent to the first layer in the neural network;
quantizing each weight of the second layer remaining after pruning based on a weighted average of a quantization and dequantization of the weight for all quantization levels to form quantized weights for the second layer;
determining the output feature maps of the second layer based on the quantized weights of the second layer;
quantizing each output feature map of the second layer based on a weighted average of a quantization and a dequantization of the output feature map for all quantization levels; and
updating parameters of the analytic threshold function, the weighted average of all quantization levels of the weights and the weighted average of each output feature map of the second layer by optimizing the cost function.
14 . The method of claim 9 , wherein the parameters of the analytic threshold function include a first parameter that controls a sharpness of the analytic threshold function, and second parameter that controls a distance between a first edge and a second edge of the analytic threshold function.
15 . A neural network analyzer, comprising:
an interface configured to receive a neural network that comprises a plurality of layers; and
a processing device configured to generate a neural network comprising at least one layer having weights and output feature maps that have been jointly pruned and quantized, to prune the weights of the at least one layer of the neural network using an analytic threshold function, to quantize each weight of the at least one layer remaining after pruning based on a weighted average of a quantization and dequantization of the weight for all quantization levels to form quantized weights for the at least one layer.
16 . The neural network analyzer of claim 15 , wherein the processing device is further configured to determine output feature maps of the at least one layer based on the quantized weights of the at least one layer, to quantize each output feature map of the at least one layer based on a weighted average of a quantization and dequantization of the output feature map for all quantization levels, and to update iteratively parameters of the analytic threshold function, the weighted average of all quantization levels of the weights and the weighted average of each output feature map of the at least one layer using a cost function.
17 . The neural network analyzer of claim 16 , wherein the interface is further configured to output the neural network comprising at least one layer having weights and output feature maps that have been jointly pruned and quantized.
18 . The neural network analyzer of claim 16 , wherein the neural network is a full-precision trained neural network before the weights and the output feature maps of the at least one layer are jointly pruned and quantized.
19 . The neural network analyzer of claim 16 , wherein the cost function includes a pruning loss term, a weight quantization loss term and a feature map quantization loss term.
20 . The neural network analyzer of claim 16 , wherein the parameters of the analytic threshold function, the weighted average of all quantization levels of the weights and the weighted average of each output feature map of the at least one layer are iteratively updated based on an optimization of the cost function, and
wherein the parameters of the analytic threshold function include a first parameter that controls a sharpness of the analytic threshold function, and second parameter that controls a distance between a first edge and a second edge of the analytic threshold function.