Stochastic bitstream generation with in-situ function mapping
View Patent ↗Techniques for generating digital outputs as stochastic bitstreams with activation function mapping are provided. In one aspect, a system includes: a shared circuitry component including a RNG for generating a sequence of random addresses to read a random sequence of digital voltage references stored in a LUT, and a DAC for converting the random sequence of digital voltage references into random analog voltage references V L ; and a comparator(s) for comparing the random analog voltage references V L and input analog voltages V N in sequences of comparisons to produce sequences of digital pulses as stochastic bitstreams. A system having multiple comparators for simultaneously comparing each of the random analog voltage references V L against more than one of the input analog voltages V N in parallel is also provided, as is a method for generating digital outputs from input analog voltages V N .
1 . A system for generating digital outputs from input analog voltages V N , the system comprising:
a shared circuitry component comprising:
a pseudo-random number generator for generating a sequence of random addresses to read a random sequence of digital voltage references selected to represent the inverse of an in-situ mapped activation function, wherein the random addresses are stored in a look-up table, and
a digital-to-analog converter for converting the random sequence of digital voltage references into random analog voltage references V L that match each random address generated by the pseudo-random number generator with a row in the look-up table containing a digital voltage reference;
an analog artificial intelligence (AI) crossbar array of nonvolatile resistive memory devices outputting the input analog voltages V N representing weighted connections between neurons in a neural network, wherein the input analog voltages V N comprise fixed input voltages for each time step of all the random analog voltage references V L ; and
a plurality of comparators for simultaneously comparing each of the random analog voltage references V L and the input analog voltages V N in sequences of comparisons performed in parallel, wherein each of the comparisons compares one of the random analog voltage references V L against one of the input analog voltages V N , and wherein each comparator produces a digital pulse as either a digital 1 or a digital 0 for each of the comparisons such that the sequences of the comparisons result in sequences of digital pulses that form stochastic bitstreams as the digital outputs that represent the neurons activations, and wherein one unique stochastic bitstream is generated for each of the input analog voltages V N .
2 . The system of claim 1 , wherein the pseudo-random number generator comprises a Linear Feedback Shift Register.
3 . The system of claim 1 , wherein the activation function is a non-linear function selected from the group consisting of: a rectified linear activation (ReLU) function, a sigmoid function, and a tanh activation function.
4 . A system for generating digital outputs from input analog voltages V N , the system comprising:
a shared circuitry component comprising:
a pseudo-random number generator for generating a sequence of random addresses to read a random sequence of digital voltage references selected to represent the inverse of an in-situ mapped activation function, wherein the random addresses are stored in a look-up table, and
a digital-to-analog converter for converting the random sequence of digital voltage references into random analog voltage references V L that match each random address generated by the pseudo-random number generator with a row in the look-up table containing a digital voltage reference;
an analog artificial intelligence (AI) crossbar array of nonvolatile resistive memory devices outputting the input analog voltages V N representing weighted connections between neurons in a neural network, wherein the input analog voltages V N comprise fixed input voltages for each time step of all the random analog voltage references V L ; and
multiple comparators for simultaneously comparing each of the random analog voltage references V L against more than one of the input analog voltages V N in sequences of comparisons performed in parallel for each of the input analog voltages V N , wherein each of the comparisons compares one of the random analog voltage references V L against one of the input analog voltages V N , wherein each of the multiple comparators produces a digital pulse as either a digital 1 or a digital 0 for each of the comparisons such that the sequences of the comparisons performed in parallel for each of the input analog voltages V N results in sequences of digital pulses that form stochastic bitstreams as the digital outputs that represent the neurons activations, and wherein one unique stochastic bitstream is generated for each of the input analog voltages V N .
5 . The system of claim 4 , wherein the input analog voltages V N comprise at least a first input analog voltage V N′ , a second input analog voltage V N″ and a third input analog voltage VN ′″ , and wherein the multiple comparators comprise at least a first comparator for comparing a given one of the random analog voltage references V L at a given time step against the first input analog voltage V N′ , a second comparator for comparing the given one of the random analog voltage references V L at the given time step against the second input analog voltage V N″ , and a third comparator for comparing the given one of the random analog voltage references V L at the given time step against the third input analog voltage V N′″ .
6 . The system of claim 4 , wherein the pseudo-random number generator comprises a Linear Feedback Shift Register.
7 . The system of claim 4 , wherein the activation function is a non-linear function selected from the group consisting of: a rectified linear activation (ReLU) function, a sigmoid function, and a tanh activation function.
8 . A method for generating digital outputs from input analog voltages V N , the method comprising:
generating, by a pseudo-random number generator in a shared circuitry component, a sequence of random addresses to read a random sequence of digital voltage references selected to represent the inverse of an in-situ mapped activation function, wherein the random addresses are stored in a look-up table;
converting, by a digital-to-analog converter in the shared circuitry component, the random sequence of digital voltage references into random analog voltage references V L that match each random address generated by the pseudo-random number generator with a row in the look-up table containing a digital voltage reference;
outputting, by an analog artificial intelligence (AI) crossbar array of nonvolatile resistive memory devices, the input analog voltages V N representing weighted connections between neurons in a neural network, wherein the input analog voltages V N comprise fixed input voltages for each time step of all the random analog voltage references V L ;
simultaneously comparing, by multiple comparators, each of the random analog voltage references V L and the input analog voltages V N in sequences of comparisons performed in parallel, wherein each of the comparisons compares one of the random analog voltage references V L against one of the input analog voltages V N ; and
producing, by the at least one comparator, a digital pulse as either a digital 1 or a digital 0 for each of the comparisons, such that the sequences of the comparisons result in sequences of digital pulses that form stochastic bitstreams as the digital outputs that represent the neurons activations, and wherein one unique stochastic bitstream is generated for each of the input analog voltages V N .
9 . The method of claim 8 , wherein the pseudo-random number generator comprises a Linear Feedback Shift Register.
10 . The method of claim 8 , further comprising:
dividing the given activation function into equally-spaced quantization levels;
determining projections from corresponding points along the given activation function to select input analog voltages V N ; and
storing digital representations of the select input analog voltages V N in the look-up table as the digital voltage references.
11 . The method of claim 10 , wherein the given activation function is a non-linear function selected from the group consisting of: a rectified linear activation (ReLU) function, a sigmoid function, and a tanh activation function.
12 . The method of claim 10 , further comprising:
selecting another activation function; and
repeating the dividing, the determining and the storing with the another activation function.
13 . The method of claim 8 , wherein multiple comparators are used to simultaneously compare each of the random analog voltage references V L against more than one of the input analog voltages V N .
14 . The method of claim 13 , wherein the input analog voltages V N comprise at least a first input analog voltage V N′ , a second input analog voltage V N″ and a third input analog voltage V N′″ , and wherein the multiple comparators comprise at least a first comparator for comparing a given one of the random analog voltage references V L at a given time step against the first input analog voltage V N′ , a second comparator for comparing the given one of the random analog voltage references V L at the given time step against the second input analog voltage V N″ , and a third comparator for comparing the given one of the random analog voltage references V L at the given time step against the third input analog voltage V N′41 .
15 . The system of claim 1 , wherein each of the random analog voltage references V L is simultaneously compared against all of the input analog voltages V N at each time step.