Hardware accelerated discretized neural network
An innovative low-bit-width device may include a first digital-to-analog converter (DAC), a second DAC, a plurality of non-volatile memory (NVM) weight arrays, one or more analog-to-digital converters (ADCs), and a neural circuit. The first DAC is configured to convert a digital input signal into an analog input signal. The second DAC is configured to convert a digital previous hidden state (PHS) signal into an analog PHS signal. NVM weight arrays are configured to compute vector matrix multiplication (VMM) arrays based on the analog input signal and the analog PHS signal. The NVM weight arrays are coupled to the first DAC and the second DAC. The one or more ADCs are coupled to the plurality of NVM weight arrays and are configured to convert the VMM arrays into digital VMM values. The neural circuit is configured to process the digital VMM values into a new hidden state.
1. A method comprising:
converting a digital input signal into an analog input signal;
converting a digital previous hidden state (PHS) signal into an analog PHS signal;
computing, using a plurality of non-volatile memory (NVM) weight arrays, a plurality of vector matrix multiplication (VMM) arrays based on the analog input signal and the analog PHS signal;
converting the VMM arrays into digital VMM values; and
processing the digital VMM values into a new hidden state.
2. The method of claim 1 , wherein processing the digital VMM values into the new hidden state further comprises:
processing the digital VMM values into a forget gate value, an input gate value, an output gate value, and a new candidate memory cell value; and
calculating the new hidden state based on the forget gate value, the input gate value, the output gate value, and the new candidate memory cell value.
3. The method of claim 1 , wherein the NVM weight arrays have a bit-width less than 32 bits.
4. The method of claim 1 , wherein the NVM weight arrays comprise resistive cross-point arrays.
5. The method of claim 1 , wherein converting the VMM arrays into the digital VMM values comprises adding an analog-to-digital conversion (ADC) noise component.
6. The method of claim 5 , wherein adding the ADC noise component comprises adding the ADC noise component before or after an ADC quantization.
7. The method of claim 1 , wherein one or more of the analog input signal, the analog PHS signal, the plurality of NVM weight arrays, and the digital VMM values are quantized to about 4 bits or less.
8. The method of claim 1 , further comprising inputting the new hidden state as the digital PHS on a subsequent iteration of the method.
9. The method of claim 1 , wherein processing the digital VMM values into the new hidden state further comprises:
calculating a new memory cell state; and
calculating the new hidden state based on the new memory cell state.
10. A device comprising:
a first digital-to-analog converter (DAC) configured to convert a digital input signal into an analog input signal;
a second DAC configured to convert a digital previous hidden state (PHS) signal into an analog PHS signal;
a plurality of non-volatile memory (NVM) weight arrays configured to compute a plurality of vector matrix multiplication (VMM) arrays based on the analog input signal and the analog PHS signal, the plurality of NVM weight arrays being coupled to the first DAC and the second DAC;
one or more analog-to-digital converters (ADCs) coupled to the plurality of NVM weight arrays, the one or more ADCs configured to convert the VMM arrays into digital VMM values; and
a neural circuit configured to process the digital VMM values into a new hidden state.
11. The device of claim 10 , wherein the plurality of NVM weight arrays comprises a plurality of resistive cross-point arrays.
12. The device of claim 10 , wherein:
an array from the plurality of NVM weight arrays includes a plurality of junctions; and
each junction of the plurality of junctions includes one or more NVM cells.
13. The device of claim 10 , wherein:
the one or more ADCs comprise a plurality of ADCs; and
the neural circuit comprises a plurality of activation components coupled to the plurality of ADCs, the plurality of activation components configured to receive and process the digital VMM values.
14. The device of claim 13 , wherein the neural circuit further comprises arithmetic circuitry coupled to the plurality of activation components, the arithmetic circuitry being configured to generate the new hidden state based on an output received from each of the plurality of activation components.
15. The device of claim 13 , further comprising a plurality of analog integrate and average components situated between the plurality of NVM weight arrays and the plurality of ADCs.
16. The device of claim 13 , further comprising a plurality of analog integrate and average components situated after the plurality of activation components.
17. The device of claim 10 , wherein the neural circuit is further configured to calculate a new memory cell state, wherein the new hidden state is generated by the neural circuit based on the new memory cell state.
18. The device of claim 10 , wherein an output of one or more of the first DAC, the second DAC, the plurality of NVM weight arrays, and the one or more ADCs is quantized to about 4 bits or less.
19. A circuit, comprising:
means for converting a digital input signal into an analog input signal;
means for converting a digital previous hidden state (PHS) signal into an analog PHS signal;
means for computing a plurality of vector matrix multiplication (VMM) arrays based on the analog input signal and the analog PHS signal;
means for converting the VMM arrays into digital VMM values; and
means for processing the digital VMM values into a new hidden state.
20. The circuit of claim 19 , wherein the means for processing the digital VMM values into the new hidden state further comprises:
means for processing the digital VMM values into a forget gate value, an input gate value, an output gate value, and a new candidate memory cell value; and
means for calculating the new hidden state based on the forget gate value, the input gate value, the output gate value, and the new candidate memory cell value.
21. The circuit of claim 19 , wherein one or more of the analog input signal, the analog PHS signal, and the digital VMM values are quantized to about 4 bits or less.
22. The circuit of claim 19 , further comprising means for inputting the new hidden state as the digital PHS on a subsequent cycle.
23. The circuit of claim 19 , wherein the means for processing the digital VMM values into the new hidden state further comprises:
means for calculating a new memory cell state; and
means for calculating the new hidden state based on the new memory cell state.
24. The circuit of claim 19 , wherein the plurality of VMM arrays is further computed using a plurality of non-volatile memory (NVM) weight arrays.