Flexible, lightweight quantized deep neural networks
To improve the throughput and energy efficiency of Deep Neural Networks (DNNs) on customized hardware, lightweight neural networks constrain the weights of DNNs to be a limited combination of powers of 2. In such networks, the multiply-accumulate operation can be replaced with a single shift operation, or two shifts and an add operation. To provide even more design flexibility, the k for each convolutional filter can be optimally chosen instead of being fixed for every filter. The present invention formulates the selection of k to be differentiable and describes model training for determining k-based weights on a per-filter basis. The present invention can achieve higher speeds as compared to lightweight NNs with only minimal accuracy degradation, while also achieving higher computational energy efficiency for ASIC implementation.
1. A method of training a deep neural network having multiple convolutional layers, each convolutional layer having one or more filters, comprising, for some or all of the one or more filters:
quantizing weights for some or all of the one or more filters as a set of numbers;
computing a residual for some or all of the one or more filters based on a difference between unquantized weights of the one or more filters and the quantized weights;
determining a parameter k for some or all of the one or more filters based on a comparison of the computed residual to a threshold;
computing a loss function for the unquantized weights and the threshold; and
updating the unquantized weights for some or all of the one or more filters based on a derivative of the computed loss function by applying a number of operations based on k for each weight.
2. The method of claim 1 , each number in the set of numbers comprising a sum of powers of 2.
3. The method of claim 1 , the number of operations comprising k shift operations and a k−1 add operations.
4. The method of claim 1 , the optimal threshold being updated based on a derivative of the computed loss function.
5. The method of claim 1 , the loss function being a sum of a cross entity loss and a regularization loss.
6. The method of claim 5 , the regularization loss being a sum of a plurality of lasso losses.
7. The method of claim 1 wherein a maximum value for k is pre-selected.
8. The method of claim 1 , the weights being quantized in accordance with the function:
Q
k
(
w
i
|
t
)
=
{
0
,
if
K
=
0
∑
j
=
0
K
-
1
(
r
i
,
j
2
>
t
j
)
R
(
k
i
,
j
)
,
if
K
≥
1
where:
K is a maximum value fork;
r i,j is a residual defined as w i −Q k (w i |t);
R(x)=sign(w)×2 [log(|x|)] ; and
[·] is a rounding to integer function.
9. The method of claim 8 , the parameter k for filter i defined as
k
i
=
∑
j
=
0
k
-
1
(
r
i
,
j
2
>
t
j
)
.
10. A system of training a deep neural network having multiple convolutional layers, each convolutional layer having one or more filters, comprising, for some or all of the one or more filters:
a processor; and
memory coupled to the processor and containing software that, when executed by the processor performs, for some or all of the one or more filters, the functions of:
quantizing weights for some or all of the one or more filters, as a set of numbers;
computing a residual for some or all of the one or more filters, based on a difference between the unquantized weights and the quantized weights;
determining a parameter k for some or all of the one or more filters, based on a comparison of the computed residual to an optimal threshold;
computing a loss function for the weights and the optimal threshold; and
updating the weights for some or all of the one or more filters, based on a derivative of the computed loss function by applying a number of operations based on k for each weight.
11. The system of claim 10 , each number in the set of numbers comprising a sum of powers of 2.
12. The system of claim 10 , the number of operations comprising k shift operations and a k−1 add operations.
13. The system of claim 10 , the optimal threshold being updated based on a derivative of the computed loss function.
14. The system of claim 10 , the loss function being a sum of a cross entity loss and a regularization loss.
15. The system of claim 14 , the regularization loss being a sum of a plurality of lasso losses.
16. The system of claim 10 wherein a maximum value for k is pre-selected.
17. The system of claim 10 , the weights being quantized in accordance with the function:
Q
k
(
w
i
❘
t
)
=
{
0
,
if
K
=
0
∑
j
=
0
K
-
1
𝕀
(
r
i
,
j
2
>
t
j
)
R
(
k
i
,
j
)
,
if
K
≥
1
where:
K is a maximum value fork;
r i,j is a residual defined as w i −Q k (w i |t);
R(x)=sign(w)×2 [log(|x|)] ; and
[·] is a rounding to integer function.
18. The system of claim 17 , the parameter k for filter i defined as
k
i
=
∑
j
=
0
k
-
1
𝕀
(
r
i
,
j
2
>
t
j
)
.