Systems and methods of sparsity exploiting
Disclosed is a neuromorphic integrated circuit including, in some embodiments, a multi-layered neural network disposed in an analog multiplier array of two-quadrant multipliers. Each multiplier of the multipliers is wired to ground and draws a negligible amount of current when input signal values for input signals to transistors of the multiplier are approximately zero, weight values of the transistors of the multiplier are approximately zero, or a combination thereof. Also disclosed is a method of the neuromorphic integrated circuit including, in some embodiments, training the neural network; tracking rates of change for the weight values; determining if and how quickly certain weight values are trending toward zero; and driving those weight values toward zero, thereby encouraging sparsity in the neural network. Sparsity in the neural network combined with the multipliers wired to ground minimizes power consumption of the neuromorphic integrated circuit such that battery power is sufficient for power.
1. A neuromorphic integrated circuit, comprising:
a multi-layered neural network disposed in an analog multiplier array of a plurality of two-quadrant multipliers arranged in a memory sector of the neuromorphic integrated circuit, wherein at least one or more of the plurality of two-quadrant multipliers is a bias-free two-quadrant multiplier,
wherein each multiplier of the multipliers is wired to ground and draws a reduced amount of current when input signal values for input signals to transistors of the multiplier are near zero or zero, weight values of the transistors of the multiplier are near zero or zero, or a combination thereof, and
wherein sparsity in the neural network combined with the number of multipliers wired to ground reduces power consumption of the neuromorphic integrated circuit.
2. The neuromorphic integrated circuit of claim 1 , wherein each multiplier of the multipliers draws no current when the input signal values for the input signals to the transistors of the multiplier are zero, the weight values of the transistors of the multiplier are zero, or a combination thereof.
3. The neuromorphic integrated circuit of claim 1 , wherein the weight values correspond to synaptic weight values between neural nodes in the neural network disposed in the neuromorphic integrated circuit.
4. The neuromorphic integrated circuit of claim 3 , wherein input signal values multiplied by the weight values provide output signal values that are combined to arrive at a decision of the neural network.
5. The neuromorphic integrated circuit of claim 1 , wherein the transistor of the two-quadrant multipliers includes a metal-oxide-semiconductor field-effect transistor (“MOSFET”).
6. The neuromorphic integrated circuit of claim 1 , wherein each bias-free two-quadrant multiplier of the two-quadrant multipliers has a differential structure configured to allow programmatic compensation for overshoot if any one of two cells is set with a higher weight value than targeted.
7. The neuromorphic integrated circuit of claim 1 , wherein the neuromorphic integrated circuit is configured for one or more application specific standard products (“ASSPs”) selected from keyword spotting, speaker identification, one or more audio filters, gesture recognition, image recognition, video object classification and segmentation, and autonomous vehicles including drones.
8. The neuromorphic integrated circuit of claim 1 , wherein the neuromorphic integrated circuit is configured to operate on battery power.
9. A method, comprising:
training a multi-layered neural network disposed in an analog multiplier array of a plurality of two-quadrant multipliers in a memory sector of the neuromorphic integrated circuit, wherein at least one or more of the plurality of two-quadrant multipliers is a bias-free two-quadrant multiplier,
wherein each multiplier of the multipliers is wired to ground and draws a first amount of current when input signal values for input signals to transistors of the multiplier are near zero or zero, weight values of the transistors of the multiplier are near zero or zero, or a combination thereof; and
encouraging sparsity in the neural network by training with a training algorithm configured to drive a plurality of the input signal values, the weight values, or the combination thereof to zero or near zero for the multipliers, thereby reducing power consumption by the neuromorphic integrated circuit.
10. The method of claim 9 , wherein:
each multiplier of the multipliers draws no current when the input signal values for the input signals to the transistors of the multiplier are zero, the weight values of the transistors of the multiplier are zero, or a combination thereof; and
each bias-free two-quadrant multiplier of the two-quadrant multipliers has a differential structure configured to allow programmatic compensation for overshoot if any one of two cells is set with a higher weight value than targeted.
11. The method of claim 9 , further comprising:
tracking rates of change for the weight values of each multiplier of the multipliers during the training; and
determining if one or more weight values are trending below a threshold or towards zero and how quickly those one or more weight values are trending below the threshold or towards zero.
12. The method of claim 11 , further comprising:
driving the weight values to zero or near zero for those one or more weight values that are trending below the threshold or towards zero during training as part of encouraging sparsity in the neural network.
13. The method of claim 11 , wherein the weight values correspond to synaptic weight values between neural nodes in the neural network of the neuromorphic integrated circuit.
14. A method, comprising:
training a multi-layered neural network disposed in an analog multiplier array of a plurality of two-quadrant multipliers in a memory sector of the neuromorphic integrated circuit, wherein at least one or more of the plurality of two-quadrant multipliers is a bias-free two-quadrant multiplier,
wherein each multiplier of the multipliers is wired to ground and draws a first amount of current when input signal values for input signals to transistors of the multiplier are near zero or zero, weight values of the transistors of the multiplier are near zero or zero, or a combination thereof;
tracking rates of change for the weight values of each multiplier of the multipliers during the training;
determining if one or more weight values are trending below a threshold or towards zero and how quickly those one or more weight values are trending below the threshold or towards zero; and
driving the weight values to zero or near zero for those one or more weight values that are trending below the threshold or towards zero, thereby encouraging sparsity in the neural network.
15. The method of claim 14 , wherein:
each multiplier of the multipliers draws no current when the input signal values for the input signals to the transistors of the multiplier are zero, the weight values of the transistors of the multiplier are zero, or a combination thereof; and
each bias-free two-quadrant multiplier of the two-quadrant multipliers has a differential structure configured to allow programmatic compensation for overshoot if any one of two cells is set with a higher weight value than targeted.
16. The method of claim 14 , further comprising:
setting a subset of the weight values to zero before training the neural network, thereby further encouraging sparsity in the neural network.
17. The method of claim 14 , wherein the training is with a training algorithm configured to drive a plurality of the input signal values, the weight values, or the combination thereof to zero or near zero for the multipliers, thereby reducing power consumption by the neuromorphic integrated circuit.
18. The method of claim 14 , wherein the training encourages sparsity in the neural network by minimizing a cost function that includes a quantity of non-zero weight values for the weight values.
19. The method of claim 14 , further comprising:
minimizing a cost function with an optimization function including gradient descent, back-propagation, or both gradient descent and back-propagation, wherein an estimate of power consumption of the neuromorphic integrated circuit is used as a component of the cost function.
20. The method of claim 14 , further comprising:
incorporating the neuromorphic integrated circuit in one or more application specific standard products (“ASSPs”) selected from keyword spotting, speaker identification, one or more audio filters, gesture recognition, image recognition, video object classification and segmentation, and autonomous vehicles including drones.