Analog hardware realization of trained neural networks
Systems and methods are provided for analog hardware realization of neural networks. The method incudes obtaining a neural network topology and weights of a trained neural network. The method also includes transforming the neural network topology to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons. The method also includes computing a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection. The method also includes generating a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value.
1 . A method for hardware realization of neural networks, comprising:
obtaining a neural network topology and weights of a trained neural network;
transforming the neural network topology into an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors, wherein each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons;
computing a weight matrix for the equivalent analog network based on the weights of the trained neural network, wherein each element of the weight matrix represents a respective connection between a respective pair of operational amplifiers of the equivalent analog network;
generating a resistance matrix for the weight matrix, wherein each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value, wherein generating the resistance matrix for the weight matrix comprises:
obtaining a predetermined range of possible resistance values {R min , R max } and selecting an initial base resistance value R base within the predetermined range;
selecting a limited length set of resistance values, within the predetermined range, that
w
i
,
j
=
R
b
a
s
e
(
1
R
i
-
1
R
j
)
within the range provides the most uniform distribution of possible weights [−R base , R base ] for all combinations of {R i , R j } within the limited length set of resistance values;
selecting a resistance value R + =R − , from the limited length set of resistance values, either for each analog neuron or for each layer of the equivalent analog network, based on maximum weight of incoming connections and bias w max of each neuron or for each layer of the equivalent analog network, such that R + =R − is the closest resistor set value to R base *w max ; and
for each element of the weight matrix, selecting a respective first resistance value R 1 and a respective second resistance value R 2 that minimizes an error according to equation
e
r
r
=
(
R
+
R
1
+
R
-
R
2
)
·
r
err
+
w
-
R
+
R
1
+
R
-
R
2
for all possible values of R 1 and R 2 within the predetermined range of possible resistance values, wherein w is the respective element of the weight matrix, and r err is a predetermined relative tolerance value for resistances; and
generating a schematic model for implementing the equivalent analog network as electronic circuitry based on the resistance matrix, wherein:
the plurality of operational amplifiers and the plurality of resistors in electronic circuitry implementing the equivalent analog network has a lower power consumption during use compared to an electronic circuit implementing the trained neural network, fewer analog components than the electronic circuit implementing the trained neural network, and/or a smaller footprint than the electronic circuit implementing the trained neural network.
2 . The method of claim 1 , wherein the predetermined range of possible resistance values includes resistances according to nominal series E24 in the range 100 KOhm to 1 MOhm.
3 . The method of claim 1 , wherein R + and R − are chosen independently for each layer of the equivalent analog network.
4 . The method of claim 1 , wherein R + and R − are chosen independently for each analog neuron of the equivalent analog network.
5 . The method of claim 1 , wherein a first one or more weights of the weight matrix and a first one or more inputs represent one or more connections to a first operational amplifier of the equivalent analog network, the method further comprising:
prior to generating the resistance matrix:
modifying the first one or more weights by a first value; and
configuring the first operational amplifier to multiply, by the first value, a linear combination of the first one or more weights and the first one or more inputs, before performing an activation function.
6 . The method of claim 1 , further comprising:
obtaining a predetermined range of weights; and
updating the weight matrix according to the predetermined range of weights such that the equivalent analog network produces similar output as the trained neural network for the same input.
7 . The method of claim 1 , wherein the trained neural network is trained so that each layer of the neural network topology has quantized weights.
8 . The method of claim 1 , further comprising:
retraining the trained neural network to reduce sensitivity to errors in the weights or the resistance values that cause the equivalent analog network to produce different output compared to the trained neural network.
9 . The method of claim 1 , further comprising:
retraining the trained neural network so as to minimize weight in any layer that are more than mean absolute weight for that layer by larger than a predetermined threshold.
10 . A system for hardware realization of neural networks, comprising:
one or more processors; and
memory;
wherein the memory stores one or more programs configured for execution by the one or more processors, and the one or more programs comprising instructions for:
obtaining a neural network topology and weights of a trained neural network;
transforming the neural network topology into an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors, wherein each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons;
computing a weight matrix for the equivalent analog network based on the weights of the trained neural network, wherein each element of the weight matrix represents a respective connection between a respective pair of operational amplifiers of the equivalent analog network;
generating a resistance matrix for the weight matrix, wherein each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value, wherein generating the resistance matrix for the weight matrix comprises:
obtaining a predetermined range of possible resistance values {R min , R max } and selecting an initial base resistance value R base within the predetermined range;
selecting a limited length set of resistance values, within the predetermined range, that provides the most uniform distribution of possible weights
w
i
,
j
=
R
b
a
s
e
(
1
R
i
-
1
R
j
)
within the range [−R base , R base ] for all combinations of {R i , R j } within the limited length set of resistance values;
selecting a resistance value R + =R − , from the limited length set of resistance values, either for each analog neuron or for each layer of the equivalent analog network, based on maximum weight of incoming connections and bias w max of each neuron or for each layer of the equivalent analog network, such that R + =R − is the closest resistor set value to R base *w max ; and
for each element of the weight matrix, selecting a respective first resistance value R 1 and a respective second resistance value R 2 that minimizes an error
e
r
r
=
(
R
+
R
1
+
R
-
R
2
)
·
r
err
+
|
w
-
R
+
R
1
+
R
-
R
2
|
for all possible values of R 1 and R 2 within the predetermined range of possible resistance values, wherein w is the respective element of the weight matrix, and r err is a predetermined relative tolerance value for resistances; and
generating a schematic model for implementing the equivalent analog network as electronic circuitry based on the resistance matrix, wherein:
the plurality of operational amplifiers and the plurality of resistors in electronic circuitry implementing the equivalent analog network has a lower power consumption during use compared to an electronic circuit implementing the trained neural network, fewer analog components than the electronic circuit implementing the trained neural network, and/or a smaller footprint than the electronic circuit implementing the trained neural network.
11 . The system of claim 10 , wherein the predetermined range of possible resistance values includes resistances according to nominal series E24 in the range 100 KOhm to 1 MOhm.
12 . The system of claim 10 , wherein R + and R − are chosen independently for each layer of the equivalent analog network.
13 . The system of claim 10 , wherein R + and R − are chosen independently for each analog neuron of the equivalent analog network.
14 . The system of claim 10 , wherein a first one or more weights of the weight matrix and a first one or more inputs represent one or more connections to a first operational amplifier of the equivalent analog network, and the one or more programs further comprise instructions for:
prior to generating the resistance matrix:
modifying the first one or more weights by a first value; and
configuring the first operational amplifier to multiply, by the first value, a linear combination of the first one or more weights and the first one or more inputs, before performing an activation function.
15 . The system of claim 10 , wherein the one or more programs further comprise instructions for:
obtaining a predetermined range of weights; and
updating the weight matrix according to the predetermined range of weights such that the equivalent analog network produces similar output as the trained neural network for the same input.
16 . The system of claim 10 , wherein the trained neural network is trained so that each layer of the neural network topology has quantized weights.
17 . The system of claim 10 , wherein the one or more programs further comprise instructions for:
retraining the trained neural network to reduce sensitivity to errors in the weights or the resistance values that cause the equivalent analog network to produce different output compared to the trained neural network.
18 . A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer system having one or more processors, the one or more programs comprising instructions for:
obtaining a neural network topology and weights of a trained neural network;
transforming the neural network topology into an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors, wherein each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons;
computing a weight matrix for the equivalent analog network based on the weights of the trained neural network, wherein each element of the weight matrix represents a respective connection between a respective pair of operational amplifiers of the equivalent analog network;
generating a resistance matrix for the weight matrix, wherein each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value, wherein generating the resistance matrix for the weight matrix comprises:
obtaining a predetermined range of possible resistance values {R min , R max } and selecting an initial base resistance value R base within the predetermined range;
selecting a limited length set of resistance values, within the predetermined range, that provides the most uniform distribution of possible weights
w
i
,
j
=
R
b
a
s
e
(
1
R
i
-
1
R
j
)
within the range [−R base , R base ] for all combinations of {R i , R j } within the limited length set of resistance values;
selecting a resistance value R + =R − , from the limited length set of resistance values, either for each analog neuron or for each layer of the equivalent analog network, based on maximum weight of incoming connections and bias w max of each neuron or for each layer of the equivalent analog network, such that R + =R − is the closest resistor set value to R base *w max ; and
for each element of the weight matrix, selecting a respective first resistance value Ry and a respective second resistance value R 2 that minimizes an error according to equation
err
=
(
R
+
R
1
+
R
-
R
2
)
·
r
err
+
❘
"\[LeftBracketingBar]"
w
-
R
+
R
1
+
R
-
R
2
❘
"\[RightBracketingBar]"
for all possible values of R 1 and R 2 within the predetermined range of possible resistance values, wherein w is the respective element of the weight matrix, and r err is a predetermined relative tolerance value for resistances; and
generating a schematic model for implementing the equivalent analog network as electronic circuitry based on the resistance matrix, wherein:
the plurality of operational amplifiers and the plurality of resistors in electronic circuitry implementing the equivalent analog network has a lower power consumption during use compared to an electronic circuit implementing the trained neural network, fewer analog components than the electronic circuit implementing the trained neural network, and/or a smaller footprint than the electronic circuit implementing the trained neural network.