Accelerating binary neural networks within latch structure of non-volatile memory devices
A non-volatile memory device includes an array of non-volatile memory cells that are configured to store weights of a neural network. Associated with the array is a data latch structure that includes a page buffer, which can store weights for a layer of the neural network that is read out of the array, and a transfer buffer, that can store inputs for the neural network. The memory device can perform multiply and accumulate operations between inputs and weight of the neural network within the latch structure, avoiding the need to transfer data out of the array and associated latch structure for portions of an inference operation. By using binary weights and inputs, multiplication can be performed by bit-wise XNOR operations. The results can then be summed and activation applied, all within the latch structure.
1. A non-volatile memory device, comprising:
an array of non-volatile memory cells configured to store at least a portion of one or more sets of weights of a neural network;
read/write circuits connected to the array of non-volatile memory cells, the read/write circuits comprising:
a read buffer configured to hold data read from the array;
a plurality of sense amplifier circuits configured to read data storing a first set of weights for a first layer of the neural network from the array into the read buffer;
an input/output interface configured to transfer data into and out of the read/write circuits; and
a transfer buffer connected to the input/output interface and configured to store a first input for the first layer of the neural network received from the input/output interface; and
one or more processing circuits configured to:
perform a first bit-wise arithmetical operation of the first set of weights of the first layer of the neural network stored in the read buffer with the first input for the first layer of the neural network stored in the transfer buffer,
sum a result of the first bit-wise arithmetical operation,
perform an activation on the sum of the result of the first bit-wise arithmetical operation, and
transfer out over the input/output interface a result of the activation on the sum of the result of the first bit-wise arithmetical operation.
2. The non-volatile memory device of claim 1 , wherein, to perform the activation, the one or more processing circuits are configured to:
determine whether the sum exceeds a threshold value.
3. The non-volatile memory device of claim 1 , wherein the read/write circuits further comprising:
an output buffer configured to store the result of the activation on the sum of the result of the first bit-wise arithmetical operation prior to being transferred out over the input/output interface.
4. The non-volatile memory device of claim 1 , wherein:
the plurality of sense amplifier circuits are further configured to read data storing a second set of weights for a first layer of the neural network from the array in the read buffer, and
the one or more processing circuits are further configured to:
perform a second bit-wise arithmetical operation the second set of weights of the first layer of the neural network stored in the read buffer with the first input for the first layer of the neural network stored in the transfer buffer;
sum a result of the second bit-wise arithmetical operation; and
perform an activation on the sum of the result of the second bit-wise arithmetical operation.
5. The non-volatile memory device of claim 4 , wherein the one or more processing circuits are further configured to:
combine a result of the activation on the sum of the result of the first bit-wise arithmetical operation and the activation on the sum of the result of the second bit-wise arithmetical operation prior to transferring out the result of the activation of the result of the first bit-wise arithmetical operation.
6. The non-volatile memory device of claim 5 , wherein:
the plurality of sense amplifier circuits are further configured to read data storing a set of weights for a second layer of the neural network from the array in the read buffer, and
the one or more processing circuits are further configured to:
store, in the transfer buffer, the accumulated result of the activation, and
perform a bit-wise arithmetical operation the set of weights of the second layer of the neural network stored in the read buffer with the accumulated result of the activation stored in the transfer buffer.
7. The non-volatile memory device of claim 1 , wherein:
the transfer buffer is further configured to store a second input for the first layer of the neural network received from the input/output interface; and
the one or more processing circuits are further configured to, prior to transferring out the result of the first bit-wise arithmetical operation, perform a bit-wise arithmetical operation the first set of weights of the first layer of the neural network stored in the read buffer with the second input for the first layer of the neural network stored in the transfer buffer.
8. The non-volatile memory device of claim 1 , further comprising:
one or more control circuits configured to perform additional activation on the result of the activation on the sum of the result of the first bit-wise arithmetical operation transferred out over the input/output interface.
9. The non-volatile memory device of claim 1 , wherein:
the array of non-volatile memory cells and the read/write circuits are formed as part of a single die.
10. The non-volatile memory device of claim 1 , wherein:
the array of non-volatile memory cells is formed as part of a first die and the read/write circuits are formed as part of a second memory die, the first die and second die forming a bonded die pair.
11. The non-volatile memory device of claim 1 , wherein the weights of the neural network are binary valued and the first input of the first layer of the neural network is binary valued.
12. The non-volatile memory device of claim 11 , wherein one or more processing circuits configured to:
perform the first bit-wise arithmetical operation the first set of weights of the first layer of the neural network with the first input for the first layer of the neural network by performing an exclusive not OR (XNOR) operation of contents of the read buffer with contents of the transfer buffer.
13. The non-volatile memory device of claim 1 , wherein the array of non-volatile memory cells is configured to store weights of a plurality of models of the neural network, the non-volatile memory device further comprising:
one or more control circuits configured to selecting a model for the neural network, the first set of weights for the first layer of the neural network corresponds to the selected model.
14. A method, comprising:
reading out data holding a first set of weights for a first layer of a neural network from a non-volatile memory array into a first buffer of a latch structure of the non-volatile memory array;
storing a first input for the first layer of the neural network in a second buffer of the latch structure; and
performing an inferencing operation for the neural network within the latch structure, comprising:
performing a first bit-wise arithmetical operation between the first set of weights for the first layer of a neural network and the first input for the first layer of the neural network;
summing a result of the first bit-wise arithmetical operation; and
performing an activation operation on the summing of the result of the first bit-wise arithmetical operation; and
transferring a result of the inferencing operation out of the latch structure.
15. The method of claim 14 , further comprising:
subsequent to performing the first bit-wise arithmetical operation, reading out data holding a second set of weights for a first layer of the neural network from the non-volatile memory array into the first buffer of the latch structure;
wherein performing the inferencing operation for the neural network within the latch structure further comprises:
performing a second bit-wise arithmetical operation between the second set of weights for the first layer of a neural network and the first input for the first layer of the neural network;
summing a result of the second bit-wise arithmetical operation; and
performing an activation operation on the summing of the result of the second bit-wise arithmetical operation.
16. The method of claim 15 , wherein performing the inferencing operation for the neural network within the latch structure further comprises:
accumulating results of the activation operation on the summing of the result of the first bit-wise arithmetical operation and the activation operation on the summing of the result of the second bit-wise arithmetical operation.
17. The method of claim 16 , further comprising:
subsequent to performing the second bit-wise arithmetical operation, reading out \ data holding a set of weights for a second layer of a neural network from a non-volatile memory array into the first buffer; and
storing the accumulated results of the activation operation of the summing of the result of the first bit-wise arithmetical operation and the activation operation on the summing of the result of the second bit-wise arithmetical operation in the second buffer,
wherein performing the inferencing operation for the neural network within the latch structure further comprises:
performing a third bit-wise arithmetical operation between the set of weights for the second layer of a neural network and the accumulated results of the activation operation of the summing of the result of the first bit-wise arithmetical operation and the activation operation on the summing of the result of the second bit-wise arithmetical operation; and
summing a result of the third bit-wise arithmetical operation.
18. The method of claim 14 , wherein the non-volatile memory array stores weights for the first layer for the neural network corresponding to a plurality of different models for the neural network, the method further comprising:
selecting a model for the neural network, wherein the first set of weights for the first layer of the neural network corresponds to the selected model.
19. A non-volatile memory device, comprising:
an array of non-volatile memory cells configured to store a plurality of weights of a neural network;
a plurality of buffers connected to the array and configured to store data read from the array; and
means for performing an inferencing operation for the neural network within the plurality of buffers by performing a first multiply and accumulation operation between weights of a first layer of the neural network read from the array into the plurality of buffers and an input for the first layer of the neural network as transferred into the plurality of buffers.
20. The non-volatile memory device of claim 19 , wherein the means for performing an inferencing operation for the neural network within the plurality of buffers further performs the inferencing operation by using a result of the first multiply and accumulation operation as an input for a second multiply and accumulation operation with weights of a second layer of the neural network read from the array into the plurality of buffers.