Low power analog circuitry for artificial neural networks
A low power analog Long Short-Term Memory (LSTM) recurrent neural network has an input layer, an array of Adaptive Filter Unit for Analog LSTM, a linear projection layer, and an output layer. The output layer has multiple nonlinear amplifiers, a nonlinear element with a sigmoidal input-output characteristic function, and a time-constant adjustable, nonlinear, low pass filter that provides the memory function of the LSTM. The LSTM memory is used with mismatch-robust weights determined by learning by computation of optimal weights values, wherein the objective function minimizes misdetection probability, and used to process a signal to detect events.
1 . A low power analog Long Short-Term Memory (LSTM) recurrent neural network signal processing device comprising:
input circuitry, and
an array, the array comprising at least one layer of a plurality of recurrent units comprising Analog Gated Recurrent Units (AGRUs), the AGRUs further comprise sigmoidal functional blocks and a low pass filter having an adjustable time constant, the array configured to receive input signals from the input circuitry;
a linear projection layer, and
an output layer;
wherein the recurrent units each comprise a vector matrix multiplier array (VMM) comprising circuitry configured to sum a plurality of products weights times inputs of the recurrent units.
2 . The low power analog LSTM recurrent neural network signal processing device of claim 1 , wherein:
the input circuitry is configured to couple at least one preprocessed input signal from sensors to the array.
3 . The low power analog LSTM recurrent neural network signal processing device of claim 2 , wherein the input circuitry comprises:
a plurality of analog bandpass filters coupled to a plurality of envelope detectors, such that a first end of a plurality of the analog bandpass filters is coupled to an input and a second end of a plurality of the analog bandpass filters is coupled to an input of an envelope detector, and outputs of the plurality of envelope detectors couple to the array.
4 . The low power analog LSTM recurrent neural network signal processing device of claim 3 , wherein the envelope detector comprises a rectifier and a current-mode low pass filter, wherein the rectifier is coupled to the current-mode low pass filter.
5 . The low power analog LSTM recurrent neural network signal processing device of claim 1 , wherein: the linear projection layer reduces a signal dimensionality of a prior layer of recurrent units by applying an adjustable weight to each output of the prior layer of recurrent units to form weight-adjusted signals, and then summing the weight-adjusted signals.
6 . The low power analog LSTM recurrent neural network signal processing device of claim 1 , wherein: the recurrent units of the array each receive a plurality of analog signals and multiply each analog signal by an adjustable low bit-width resolution weight value.
7 . The low power analog LSTM recurrent neural network signal processing device of claim 6 , wherein:
each low bit-width resolution weight is ternarized to one of {−1, 0, 1}.
8 . The low power analog LSTM recurrent neural network of claim 2 , wherein the input circuitry comprises a plurality of pyknogram filters.
9 . The low power analog LSTM recurrent neural network of claim 8 , wherein each pyknogram filter comprises an adaptive bandpass filter having a center frequency, the adaptive bandpass filter configured to track a frequency of a strongest signal within a frequency range.
10 . A low power analog Long Short-Term Memory (LSTM) recurrent neural network signal processing device comprising:
input circuitry, and
an array, the array comprising at least one layer of a plurality of recurrent units comprising Adaptive Filter Unit for Analog LSTM (AFUA) where each AFUA recurrent unit cell comprises a ternary-weight multiply-accumulate circuitry coupled through a softmax function to an analog low-pass filter, the array configured to receive input signals from the input circuitry;
a linear projection layer, and
an output layer;
wherein the recurrent units each comprise a vector matrix multiplier array (VMM) comprising circuitry configured to sum a plurality of products weights times inputs of the recurrent units.
11 . A method for detecting events, comprising:
using a method comprising learning mismatch-robust weights learning for a low power analog LSTM recurrent neural network signal processing device comprising: computation of optimal weights values, wherein an objective function is to minimize a misdetection probability;
where the LSTM recurrent neural network comprises: an array comprising a plurality of Adaptive Filter Unit for Analog LSTM (AFUAs) analog recurrent units, a linear projection layer, and an output layer;
and operating the LSTM recurrent neural network with the computed optimal weights values to process a signal to detect the events; and
wherein the AFUAs comprise a low pass filter array and a vector matrix multiplier array (VMM) comprising circuitry configured to sum a plurality of products of multiplying weights times inputs of the AFUAs.
12 . A method of claim 11 , wherein the computed optimal weights values are computed using a multi-iteration Monte Carlo Backpropagation method adapted to minimize effects of component mismatch in the AFUAs.
13 . A method of claim 12 , wherein the multi-iteration Monte Carlo Backpropagation method comprises:
using statistical parameters applied to a transistor-level description of the LSTM recurrent neural network's activation functions and mathematical operators to simulate matching and process variations to give a probability distribution of activation function parameters and generating perturbed network models; and
perturbing activation functions of the LSTM neural network from the probability distributions of activation function parameters.
14 . A method of claim 11 , further comprising:
generating a set of perturbed networks from statistical models on fundamental device-level mismatch, and
iteratively computing activation function parameters for each of the set of perturbed networks, wherein low-precision weights are used during forward pass calculations and full-precision weights are used during backpropagation pass calculations.
15 . A method of detecting spoken keywords comprising:
filtering audio to determine a spectral analysis of an audio signal; and
providing the spectral analysis of the audio signal to the method for detecting events of claim 11 .
16 . The low power analog LSTM recurrent neural network signal processing device of claim 10 , wherein: the input circuitry is configured to couple at least one preprocessed input signal from sensors to the array.
17 . The low power analog LSTM recurrent neural network of claim 16 , wherein the input circuitry comprises a plurality of pyknogram filters.
18 . The low power analog LSTM recurrent neural network of claim 17 , wherein each pyknogram filter comprises an adaptive bandpass filter having a center frequency, the adaptive bandpass filter configured to track a frequency of a strongest signal within a frequency range.
19 . The low power analog LSTM recurrent neural network signal processing device of claim 10 , wherein the input circuitry comprises:
a plurality of analog bandpass filters coupled to a plurality of envelope detectors, such that a first end of a plurality of the analog bandpass filters is coupled to an input and a second end of a plurality of the analog bandpass filters is coupled to an input of an envelope detector, and outputs of the plurality of envelope detectors couple to the array.
20 . The low power analog LSTM recurrent neural network signal processing device of claim 19 , wherein the envelope detector comprises a rectifier and a current-mode low pass filter, wherein the rectifier is coupled to the current-mode low pass filter.