Hardware accelerated discretized neural network
An innovative low-bit-width device may include a first digital-to-analog converter (DAC), a second DAC, a plurality of non-volatile memory (NVM) weight arrays, one or more analog-to-digital converters (ADCs), and a neural circuit. The first DAC is configured to convert a digital input signal into an analog input signal. The second DAC is configured to convert a digital previous hidden state (PHS) signal into an analog PHS signal. NVM weight arrays are configured to compute vector matrix multiplication (VMM) arrays based on the analog input signal and the analog PHS signal. The NVM weight arrays are coupled to the first DAC and the second DAC. The one or more ADCs are coupled to the plurality of NVM weight arrays and are configured to convert the VMM arrays into digital VMM values. The neural circuit is configured to process the digital VMM values into a new hidden state.
1. A method comprising:
converting a digital input signal into an analog input signal;
computing, using a plurality of non-volatile memory (NVM) weight arrays, a plurality of vector matrix multiplication (VMM) arrays based on the analog input signal, wherein multiple parallel NVM cells in the plurality of NVM weight arrays represent one synaptic weight element;
converting the VMM arrays into digital VMM values;
processing the digital VMM values through at least one activation function unit, wherein processing the digital VMM values calculates a new memory cell state; and
feeding the new memory cell state as an input for processing the digital VMM values during a next cycle.
2. The method of claim 1 , wherein processing the digital VMM values through at least one activation function unit comprises:
processing the digital VMM values into a forget gate value, an input gate value, an output gate value, and a new candidate memory cell value;
calculating a hidden state based on the forget gate value, the input gate value, the output gate value, and the new candidate memory cell value; and
feeding the hidden state as an input for computing the plurality of VMM arrays during the next cycle.
3. The method of claim 2 , wherein calculating the hidden state is further based on the new memory cell state of a current cycle.
4. The method of claim 1 , wherein the multiple parallel NVM cells include at least three parallel NVM cells per one synaptic weight element.
5. The method of claim 1 , further comprising:
averaging, before ADC quantization, redundant runs of the plurality of VMM arrays.
6. The method of claim 5 , wherein at least three redundant runs are used for averaging redundant runs of the plurality of VMM arrays.
7. The method of claim 5 , wherein at least five redundant runs are used for averaging redundant runs of the plurality of VMM arrays.
8. The method of claim 1 , further comprising:
averaging redundant runs of a plurality of activation function units to determine a plurality of activation function unit values; and
processing the plurality of activation function unit values through element-wise calculations to calculate a hidden state.
9. The method of claim 1 , wherein the NVM weight arrays comprise resistive cross-point arrays.
10. A device comprising:
at least one digital-to-analog converter (DAC) configured to convert a digital input signal into an analog input signal;
a plurality of non-volatile memory (NVM) weight arrays configured to compute a plurality of vector matrix multiplication (VMM) arrays based on the analog input signal, wherein:
the plurality of NVM weight arrays is coupled to the at least one DAC; and
multiple parallel NVM cells in the plurality of NVM weight arrays represent one synaptic weight element;
at least one analog-to-digital converter (ADC) coupled to the plurality of NVM weight arrays, wherein the at least one ADC is configured to convert the VMM arrays into digital VMM values; and
a neural circuit configured to:
process the digital VMM values through at least one activation function unit, wherein processing the digital VMM values calculates a new memory cell state; and
feed the new memory cell state as an input for processing the digital VMM values during a next cycle.
11. The device of claim 10 , wherein the plurality of NVM weight arrays comprises a plurality of resistive cross-point arrays.
12. The device of claim 10 , wherein:
at least one array from the plurality of NVM weight arrays includes a plurality of junctions; and
each junction of the plurality of junctions includes multiple parallel NVM cells.
13. The device of claim 10 , wherein the multiple parallel NVM cells include at least three parallel NVM cells per one synaptic weight element.
14. The device of claim 10 , wherein:
the at least one ADC comprise a plurality of ADCs;
the neural circuit comprises a plurality of activation function units coupled to the plurality of ADCs; and
the plurality of activation function units is configured to receive and process the digital VMM values.
15. The device of claim 14 , wherein:
the neural circuit further comprises arithmetic circuitry coupled to the plurality of activation function units; and
the arithmetic circuitry is configured to generate a hidden state based on an output received from each activation function unit of the plurality of activation function units.
16. The device of claim 15 , further comprising:
at least one averaging component within the neural circuit and configured to average redundant runs of each activation function unit of the plurality of activation function units to determine a plurality of activation function unit values as output to the arithmetic circuitry.
17. The device of claim 10 , further comprising:
at least one analog integrate and average component situated between the plurality of NVM weight arrays and the at least one ADC and configured to average, before ADC quantization, redundant runs of the plurality of VMM arrays.
18. The device of claim 17 , wherein the at least one analog integrate and average component is configured to average at least three redundant runs for each VMM array of the plurality of VMM arrays.
19. The device of claim 17 , wherein the at least one analog integrate and average component is configured to average at least five redundant runs for each VMM array of the plurality of VMM arrays.
20. A circuit, comprising:
means for converting a digital input signal into an analog input signal;
means for computing, using a plurality of non-volatile memory (NVM) weight arrays, a plurality of vector matrix multiplication (VMM) arrays based on the analog input signal;
means for converting the VMM arrays into digital VMM values;
means for processing the digital VMM values through at least one activation function unit to calculate a new memory cell state; and
means for feeding the new memory cell state as an input for processing the digital VMM values during a next cycle.