Systems and methods for modifying neural networks for binary processing applications
In one embodiment, a method of simulating an operation of an artificial neural network on a binary neural network processor includes receiving a binary input vector for a layer including a probabilistic binary weight matrix and performing vector-matrix multiplication of the input vector with the probabilistic binary weight matrix, wherein the multiplication results are modified by simulated binary-neural-processing hardware noise, to generate a binary output vector, where the simulation is performed in the forward pass of a training algorithm for a neural network model for the binary-neural-processing hardware.
1. A method of training an artificial neural network based on simulated properties of a binary neural network processor, comprising:
receiving a binary input vector for a layer of the artificial neural network comprising a probabilistic binary weight matrix;
generating simulated binary-neural-processing hardware noise based on permutations of properties of the binary neural network processor;
performing vector-matrix multiplication of the input vector with the probabilistic binary weight matrix;
modifying the multiplication results based on the simulated binary-neural-processing hardware noise, to generate a binary output vector; and
forward propagating the binary output vector to one or more other layers of the artificial neural network.
2. The method of claim 1 , wherein the binary neural network processor is a compute-in-memory (CIM) device.
3. The method of claim 1 , wherein the probabilistic binary weight matrix comprises probabilities that the weights are 0 or 1.
4. The method of claim 1 , wherein the simulated binary-neural-processing hardware noise simulates process, voltage, and temperature (PVT) variations of the simulated binary-neural-processing hardware.
5. The method of claim 4 , wherein the thermal noise is varied for each multiplication.
6. The method of claim 4 , wherein:
the PVT variation noise includes capacitor variation noise;
generating the binary output vector comprises calculating a population count; and
capacitor variation noise depends on the population count.
7. The method of claim 1 , wherein:
generating the binary output vector comprises calculating a population count with noise ã;
ã=αa+β+γ;
a is a clean population count;
α represents capacitor variation noise;
β represents offset variation noise; and
γ represents thermal noise.
8. The method of claim 1 , wherein:
the layer is larger than a processing array of the binary neural network processor;
the binary neural network processor comprises analog-to-digital converters (ADCs);
performing the vector-matrix multiplication comprises splitting the layer and generating intermediate results digitized by the ADCs; and
the simulated binary-neural-processing hardware noise includes quantizing noise from digitizing the intermediate results by the ADCs.
9. The method of claim 1 , wherein:
performing the vector-matrix multiplication comprises using the binary neural network process in a sharing configuration; and
the simulated binary-neural-processing hardware noise includes correlated noise for the sharing configuration.
10. The method of claim 1 , wherein the simulated binary-neural-processing hardware noise is generated from a low-level circuit simulation of the binary neural network processor.
11. The method of claim 1 , wherein:
the artificial neural network comprises a first layer and a plurality of additional layers;
the probabilistic binary weight matrix may correspond to any of the plurality of additional layers; and
the probabilistic binary weight matrix may not correspond to the first layer.
12. An apparatus comprising a processor and a memory, the apparatus configured to train an artificial neural network based on simulated properties of a binary neural network processor, the training comprising:
receiving a binary input vector for a layer of the artificial neural network comprising a probabilistic binary weight matrix;
generating simulated binary-neural-processing hardware noise based on permutations of properties of the binary neural network processor;
performing vector-matrix multiplication of the input vector with the probabilistic binary weight matrix;
modifying the multiplication results based on the simulated binary-neural-processing hardware noise, to generate a binary output vector; and
forward propagating the binary output vector to one or more other layers of the artificial neural network.
13. The apparatus of claim 12 , wherein the binary neural network processor is a compute-in-memory (CIM) device.
14. The apparatus of claim 12 , wherein the probabilistic binary weight matrix comprises probabilities that the weights are 0 or 1.
15. The apparatus of claim 12 , wherein the simulated binary-neural-processing hardware noise simulates process, voltage, and temperature (PVT) variations of the simulated binary-neural-processing hardware.
16. The apparatus of claim 15 , wherein the thermal noise is varied for each multiplication.
17. The apparatus of claim 15 , wherein:
the PVT variation noise includes capacitor variation noise;
generating the binary output vector comprises calculating a population count; and
capacitor variation noise depends on the population count.
18. The apparatus of claim 12 , wherein:
generating the binary output vector comprises calculating a population count with noise ã;
ã=αa+β+γ;
a is a clean population count;
α represents capacitor variation noise;
β represents offset variation noise; and
γ represents thermal noise.
19. The apparatus of claim 12 , wherein:
the layer is larger than a processing array of the binary neural network processor;
the binary neural network processor comprises analog-to-digital converters (ADCs);
performing the vector-matrix multiplication comprises splitting the layer and generating intermediate results digitized by the ADCs; and
the simulated binary-neural-processing hardware noise includes quantizing noise from digitizing the intermediate results by the ADCs.
20. The apparatus of claim 12 , wherein:
performing the vector-matrix multiplication comprises using the binary neural network process in a sharing configuration; and
the simulated binary-neural-processing hardware noise includes correlated noise for the sharing configuration.
21. The apparatus of claim 12 , wherein the simulated binary-neural-processing hardware noise is generated from a low-level circuit simulation of the binary neural network processor.
22. The apparatus of claim 12 , wherein:
the artificial neural network comprises a first layer and a plurality of additional layers;
the probabilistic binary weight matrix may correspond to any of the plurality of additional layers; and
the probabilistic binary weight matrix may not correspond to the first layer.
23. A non-transitory computer-readable medium storing computer-executable code for operation of a neural network, comprising code to train an artificial neural network based on simulated properties of a binary neural network processor, the training comprising:
receiving a binary input vector for a layer of the artificial neural network comprising a probabilistic binary weight matrix;
generating simulated binary-neural-processing hardware noise based on permutations of properties of the binary neural network processor;
performing vector-matrix multiplication of the input vector with the probabilistic binary weight matrix;
modifying the multiplication results based on the simulated binary-neural-processing hardware noise, to generate a binary output vector; and
forward propagating the binary output vector to one or more other layers of the artificial neural network.