Dropout in neutral networks using threshold switching selectors in non-volatile memories
A non-volatile memory device is configured for in-memory computation of layers of a neural network by storing weight values as conductance values in memory cells formed of a series combination of a threshold switching selector, such as an ovonic threshold switch, and a programmable resistive element, such as a ReRAM element. By scaling the input voltages (representing inputs for the layer of the neural network) relative to the threshold values of the threshold switching selectors, dropout for inputs can be implemented to reduce overfitting by the neural network.
1. A non-volatile memory device, comprising:
a control circuit configured to connect to one or more arrays of non-volatile memory cells connected along word lines and bit lines, each memory cell comprising a programmable resistive element connected in series with a threshold switching selector configured to become conductive in response to application of a voltage level exceeding a threshold voltage, the control circuit is configured to:
receive a set of input values for a first layer of a neural network;
scale the set of input values relative to the threshold voltage of the threshold switching selectors;
convert the set of scaled input values to a corresponding set of scaled input voltages;
perform an inferencing operation for the first layer of the neural network by applying the scaled set of input voltages to word lines of a first of the arrays storing weight values of the first layer of the neural network in the memory cells thereof; and
determine a set of output values for the first layer of the neural network from a corresponding set of output currents on the bit lines of the first array.
2. The non-volatile memory device of claim 1 , wherein the control circuit is formed on a control die, the non-volatile memory device further comprising:
a memory die including the one or more of the arrays of non-volatile memory cells, the memory die formed separately from and bonded to the control die.
3. The non-volatile memory device of claim 1 , wherein the control circuit is further configured to:
receive the weight values of the first layer of the neural network; and
program the weight values of the first layer of the neural network in the memory cells of the first array.
4. The non-volatile memory device of claim 1 , wherein the control circuit is further configured to:
reprogram one or more of the weight values of the first layer of the neural network in the memory cells of the first array based on the set of output values of the first layer.
5. The non-volatile memory device of claim 1 , the control circuit comprising:
a plurality of current-steering circuits each configured to receive a corresponding one of the set of output currents, scale the corresponding output current, and provide the corresponding scaled output current as a corresponding one of the set of output values for the first layer of the neural network,
wherein the control circuit is further configured to:
provide the set of output values for the first layer of the neural network as inputs to a second layer of the neural network.
6. The non-volatile memory device of claim 1 , wherein the control circuit is further configured to:
digitize the set of output currents to determine a digitized value for each of the output currents;
determine the set of output values for the first layer of the neural network by truncating a number of least significant bits from the digitized values of the output currents; and
provide the set of output values for the first layer of the neural network as inputs to a second layer of the neural network.
7. The non-volatile memory device of claim 1 , wherein the input values for the first layer of the neural network and the weight values for the first layer of the neural network are analog values.
8. The non-volatile memory device of claim 1 , further comprising:
the one or more arrays of non-volatile memory cells, wherein the programmable resistive element comprises a resistive random-access memory (ReRAM) device.
9. The non-volatile memory device of claim 1 , further comprising:
the one or more arrays of non-volatile memory cells, wherein the programmable resistive element comprises a magnetic random-access memory (MRAM) device.
10. The non-volatile memory device of claim 1 , further comprising:
the one or more arrays of non-volatile memory cells, wherein the threshold switching selector comprises an ovonic threshold switch (OTS).
11. A method, comprising:
receiving a set of input values for a first layer of a neural network at an array of a plurality of non-volatile memory cells each comprising a programmable resistive element connected in series with a threshold switching selector, the programmable resistive element storing a weight value of the first layer of the neural network and the threshold switching selector configured to become conductive in response to application of a voltage level exceeding a threshold voltage;
scaling the set of input values relative to the threshold voltage of the threshold switching selectors;
converting the set of scaled input values to a corresponding set of scaled input voltages;
performing an inferencing operation for the first layer of the neural network by applying the scaled set of input voltages to the first of the array; and
determining a set of output values for the first layer of the neural network from a corresponding set of output currents on bit lines of the array.
12. The method of claim 11 , further comprising:
reprogramming one or more of the weight values of the first layer of the neural network in the memory cells of the array based on the set of output values of the first layer.
13. The method of claim 11 , wherein the input values for the first layer of the neural network are digital values and scaling the set of input values relative to the threshold voltage of the threshold switching selectors comprises:
truncating a number of least significant bits from the digital values of the input values for the first layer of the neural network.
14. The method of claim 11 , wherein the input values for the first layer of the neural network are analog current values and scaling the set of input values relative to the threshold voltage of the threshold switching selectors comprises:
scaling analog current values in a current mirror.
15. The method of claim 11 , further comprising:
prior to receiving the set of input values for a first layer of the neural network, programming the weight values of the first layer of the neural network in the memory cells of the array.
16. The method of claim 13 , further comprising:
providing the set of output values for the first layer of the neural network as inputs to a second layer of the neural network.
17. The method of claim 14 , further comprising:
providing the set of output values for the first layer of the neural network as inputs to a second layer of the neural network.
18. The method of claim 11 , wherein the input values for the first layer of the neural network and the weight values for the first layer of the neural network are analog values.
19. The method of claim 11 , wherein the programmable resistive element comprises a magnetic random-access memory (MRAM) device.
20. The method of claim 11 , wherein the threshold switching selector comprises an ovonic threshold switch (OTS).