Analog hardware implementation of activation functions
An analog neural network including a hardware activation function is provided. A layer of the analog neural network includes a sequence of processing elements that receives analog signals, perform MAC operations on the analog signals, and generates analog outputs. The analog outputs are provided to an analog circuitry that can apply an activation function on the analog outputs. The output of the analog circuitry are also analog signals, which can further be provided to the next layer in the network. The analog circuitry may include a differential pair of transistors to compute the tanh activation function. Alternatively, the analog circuitry may include a comparator and multiplexer to compute the ReLU activation function. Compared with digital implementation of activation functions, the analog circuitry eliminates the need of converting the analog outputs of the layer to digital signals and the need of converting the result of the activation function to analog signals.
1 . An analog neural network, comprising:
a plurality of processing elements to:
receive a first input comprising a first analog signal, and
perform multiplication and accumulation operations on the first input to generate a first output, the first output comprising a second analog signal; and
an analog circuit comprising a first transistor and a second transistor, the analog circuit to:
receive a second input comprising the second analog signal, and
compute an activation function with the second analog signal to generate a second output, the second output comprising a third analog signal,
wherein:
the first transistor is a first bipolar junction transistor comprising a first collector, a first emitter, and a first base,
the second transistor is a second bipolar junction transistor comprising a second collector, a second emitter, and a second base,
the first emitter and the second emitter are coupled to a current source,
each of the first collector and the second collectors is coupled to a resistor, and
the first base or the second base is configured to receive the second analog signal.
2 . The analog neural network of claim 1 , wherein the activation function is a hyperbolic tangent activation function, and the analog circuit is configured to compute the activation function with the second analog signal by converting values in the second analog signal into values in a predetermined range.
3 . The analog neural network of claim 1 , wherein the first transistor is coupled to a first resistor, and the second transistor is coupled to a second resistor.
4 . The analog neural network of claim 3 , wherein a resistance of the first resistor is the same as a resistance of the second resistor.
5 . The analog neural network of claim 1 , wherein the second input further comprises a fourth analog signal in addition to the second analog signal, the fourth analog signal and the second analog signal are differential signals, the first base is configured to receive the second analog signal, and the second base is configured to receive the fourth analog signal.
6 . The analog neural network of claim 1 , wherein a size of the first transistor is the same as a size of the second transistor.
7 . The analog neural network of claim 1 , wherein:
the analog circuit comprises a first group of transistors and a second group of transistors,
the transistors in the first group are in parallel,
the transistors in the second group are in parallel, and
a centroid of the first group of transistors matches a centroid of the second group of transistors.
8 . The analog neural network of claim 7 , wherein the transistors in the first group and the transistors in the second group are all identical.
9 . The analog neural network of claim 1 , wherein the activation function is a rectified linear unit activation function, and the analog circuit is configured to apply the activation function on the second analog signal by converting one or more values in the second analog signal into a predetermined value and converting one or more other values in the second analog signal into zero.
10 . The analog neural network of claim 1 , wherein the analog circuit comprises an analog comparator configured to receive the second analog signal and an analog multiplexer configured to output the third analog signal.
11 . The analog neural network of claim 1 , wherein:
the second output further comprises a fourth analog signal in addition to the third analog signal,
the analog circuit comprises an analog comparator, a first analog multiplexer, and a second analog multiplexer,
the analog comparator configured to receive the second analog signal and the fourth analog signal,
the first analog multiplexer is configured to output the third analog signal, and
the second analog multiplexer is configured to output the fourth analog signal.
12 . The analog neural network of claim 1 , wherein the plurality of processing elements constitutes a layer of the analog neural network, wherein the analog neural network further comprises an additional layer that is configured to:
receive the third analog signal, and
perform multiplication and accumulation operations on the third analog signal.
13 . The analog neural network of claim 1 , wherein a processing element is a memory device.
14 . The analog neural network of claim 1 , wherein the plurality of processing elements is arranged in columns and rows.
15 . One or more non-transitory computer-readable media storing instructions executable to perform operations for deep learning, the operations comprising:
inputting a first analog signal into a first layer of an analog neural network, the first layer configured to perform multiplication operations and accumulation operations on the first input to generate a second analog signal;
applying an activation function on the second analog signal by using an analog circuit, the analog circuit configured to output a third analog signal, the analog circuit comprising a group of transistors, the group of transistors comprising a first subset of one or more transistors and a second subset of one or more transistors, a centroid of the first subset matching a centroid of the second subset, wherein applying the activation function comprises inputting the second analog signal into a transistor in the group; and
inputting the third analog signal into a second layer of the analog neural network.
16 . The one or more non-transitory computer-readable media of claim 15 , wherein the analog circuit includes an analog comparator and an analog multiplexer, and applying the activation function on the second analog signal by using the analog circuit comprises inputting the second analog signal into the analog comparator.
17 . An analog neural network, comprising:
a plurality of processing elements to:
receive a first input comprising a first analog signal, and
perform multiplication and accumulation operations on the first input to generate a first output, the first output comprising a second analog signal; and
an analog circuit comprising a first transistor and a second transistor, the analog circuit to:
receive a second input comprising the second analog signal, and
compute an activation function with the second analog signal to generate a second output, the second output comprising a third analog signal,
wherein:
the first transistor is a first metal-oxide-semiconductor field-effect transistor (MOSFET) transistor comprising a first source region, a first drain region, and a first gate,
the second transistor is a second MOSFET transistor comprising a second source region a second drain region, and a second gate,
the first source region and the second source region are coupled to a current source,
each of the first drain region and the second drain region is coupled to a resistor, and
the first gate or the second gate is configured to receive the second input.
18 . The analog neural network of claim 17 , wherein the second input further comprises a fourth analog signal in addition to the second analog signal, the fourth analog signal and the second analog signal are differential signals, the first gate is configured to receive the second analog signal, and the second gate is configured to receive the fourth analog signal.
19 . The analog neural network of claim 17 , wherein the activation function is a hyperbolic tangent activation function, wherein the analog circuit is configured to compute the activation function with the second analog signal by converting values in the second analog signal into values in a predetermined range.
20 . The analog neural network of claim 17 , wherein the first transistor is coupled to a first resistor and the second transistor is coupled to a second resistor, wherein a resistance of the first resistor is the same as a resistance of the second resistor.
21 . The analog neural network of claim 17 , wherein a size of the first transistor is the same as a size of the second transistor.
22 . The analog neural network of claim 17 , wherein the analog circuit further comprises:
an analog comparator to receive the second analog signal; and
an analog multiplexer to output the third analog signal.
23 . The analog neural network of claim 17 , wherein:
the second output further comprises a fourth analog signal in addition to the third analog signal,
the analog circuit comprises an analog comparator, a first analog multiplexer, and a second analog multiplexer,
the analog comparator configured to receive the second input,
the first analog multiplexer is configured to output the third analog signal, and
the second analog multiplexer is configured to output the fourth analog signal.
24 . The analog neural network of claim 17 , wherein the plurality of processing elements constitutes a layer of the analog neural network, wherein the analog neural network further comprises an additional layer, the additional layer to receive the third analog signal and to perform multiplication and accumulation operations on the third analog signal.
25 . The analog neural network of claim 17 , wherein a processing element is a memory device.