Non-uniform quantization for flexible power-of-two computations in neural networks
Devices, systems, and methods for improving operation of a memory device that uses a deep neural network (DNN), based on using non-uniform quantization for flexible power-of-two computations, are described. An example method includes receiving a plurality of initial weights of the DNN configured to determine a value of a read voltage associated with memory device, wherein the plurality of initial weights comprises at least one non-power-of-two quantized value or at least one floating point value. The method then aggregates the plurality of initial weights to generate a plurality of quantization functions, determines each of a plurality of quantized weights for a corresponding one of the plurality of quantization functions such that each of the plurality of quantized weights is a sum of powers-of-two, and configures the DNN to use the plurality of quantized weights to generate an updated value of the read voltage for retrieving information from the memory device.
1 . A method for operating a memory device that uses a deep neural network, comprising:
receiving a plurality of initial weights of the deep neural network (DNN) trained to determine a value of a read voltage associated with the memory device, wherein the plurality of initial weights comprises at least one non-power-of-two quantized value or at least one floating point value;
determining, based on aggregating the plurality of initial weights, each of a plurality of quantized weights as a sum of powers-of-two;
configuring the DNN to use the plurality of quantized weights; and
using the DNN to enable generation of an updated value of the read voltage for retrieving information from the memory device, wherein inputs to the DNN comprise a checksum or a ones count associated with a previous value of the read voltage.
2 . The method of claim 1 , wherein a quantized weight of the plurality of quantized weights is determined based on (1) a number of power-of-two terms in the quantized weight (r), (2) a parameter indicative of a precision of the quantized weight (k), and (3) a quantization level for the quantized weight (l), and wherein each of r, k, and l is a positive integer.
3 . The method of claim 2 , wherein the quantized weight(s) is an element of a set of quantized weights(S), wherein the quantized weight is determined as:
s
∈
∑
i
=
0
r
-
1
p
i
where
p
i
∈
{
0
,
2
-
1
,
2
-
2
,
…
,
2
-
2
k
+
2
}
,
and wherein a cardinality of the set of quantized weights (|S|) is equal to the quantization level (l) for the quantized weight.
4 . The method of claim 2 , wherein the quantization level for the quantized weight is greater than or equal to 8.
5 . The method of claim 4 , wherein the number of power-of-two terms in the quantized weight is equal to 4, the parameter indicative of the precision of the quantized weight is equal to 4, and the quantization level for the quantized weight is equal to 8.
6 . The method of claim 1 , wherein aggregating the plurality of initial weights is performed using a clustering algorithm that comprises a Llyod-Max clustering algorithm or a k-means clustering algorithm.
7 . The method of claim 1 , further comprising:
scaling, prior to configuring the DNN, each of the plurality of initial weights by a first scaling factor such that a maximum absolute value of each of the plurality of initial weights is less than or equal to 1.
8 . The method of claim 7 , wherein using the DNN comprises performing a multiply-accumulate operation based on at least one of the plurality of quantized weights, and wherein the method further comprises:
scaling an output of the multiply-accumulate operation by a second scaling factor,
wherein the first scaling factor is 1/β and the second scaling factor is β.
9 . The method of claim 1 , wherein the plurality of quantized weights are stored in a compressed format that is associated with a look-up table.
10 . The method of claim 9 , wherein the compressed format comprises an n-bit index that is an address for a row in the look-up table that stores a corresponding quantized weight.
11 . The method of claim 1 , wherein the memory device is a triple level cell (TLC) NAND flash memory device.
12 . The method of claim 1 , wherein the plurality of quantized weights are determined based on a weight distribution in at least one algorithm implemented by the deep neural network.
13 . The method of claim 12 , wherein the at least one algorithm is a double backpropagation (DBP) algorithm.
14 . A system for operating a memory device that uses a deep neural network (DNN), comprising:
a processor and a memory including instructions stored thereupon, wherein the instructions upon execution by the processor cause the processor to:
receive a plurality of initial weights of the DNN trained to determine a value of a read voltage associated with the memory device, wherein the plurality of initial weights comprises at least one non-power-of-two quantized value or at least one floating point value;
determine, based on aggregating the plurality of initial weights, each of a plurality of quantized weights as a sum of powers-of-two;
configure the DNN to use the plurality of quantized weights; and
use the DNN to enable generation of an updated value of the read voltage for retrieving information from the memory device, wherein inputs to the DNN comprise a checksum or a ones count associated with a previous value of the read voltage.
15 . The system of claim 14 , wherein a quantized weight of the plurality of quantized weights is determined based on (1) a number of power-of-two terms in the quantized weight (r), (2) a parameter indicative of a precision of the quantized weight (k), and (3) a quantization level for the quantized weight (l), and wherein each of r, k, and l is a positive integer.
16 . The system of claim 14 , further comprising:
scaling, prior to configuring the DNN, each of the plurality of quantized weights by a scaling factor such that a maximum absolute value of each of the plurality of quantized weights is less than or equal to 1.
17 . The system of claim 14 , wherein the plurality of quantized weights are stored in a compressed format that is associated with a look-up table, and wherein the compressed format comprises an n-bit index that is an address for a row in the look-up table that stores a corresponding quantized weight.
18 . A non-transitory computer-readable storage medium having instructions stored thereupon for operating a memory device that uses a deep neural network, comprising:
instructions for receiving a plurality of initial weights of the deep neural network (DNN) trained to determine a value of a read voltage associated with the memory device, wherein the plurality of initial weights comprises at least one non-power-of-two quantized value or at least one floating point value;
instructions for determining, based on aggregating the plurality of initial weights, each of a plurality of quantized weights as a sum of powers-of-two;
instructions for configuring the DNN to use the plurality of quantized weights; and
instructions for using the DNN to enable generation of an updated value of the read voltage for retrieving information from the memory device, wherein inputs to the DNN comprise a checksum or a ones count associated with a previous value of the read voltage.
19 . The non-transitory computer-readable storage medium of claim 18 , wherein a quantized weight of the plurality of quantized weights is determined based on (1) a number of power-of-two terms in the quantized weight (r), (2) a parameter indicative of a precision of the quantized weight (k), and (3) a quantization level for the quantized weight (l), and wherein each of r, k, and l is a positive integer.
20 . The non-transitory computer-readable storage medium of claim 18 , wherein the plurality of quantized weights are stored in a compressed format that is associated with a look-up table, and wherein the compressed format comprises an n-bit index that is an address for a row in the look-up table that stores a corresponding quantized weight.