Structured weight based sparsity in an artificial neural network compiler
View Patent ↗A novel and useful system and method of improved power performance and lowered memory requirements for an artificial neural network based on packing memory utilizing several structured sparsity mechanisms. The invention applies to neural network (NN) processing engines adapted to implement mechanisms to search for structured sparsity in weights and activations, resulting in a considerably reduced memory usage. The sparsity guided training mechanism synthesizes and generates structured sparsity weights. A compiler mechanism within a software development kit (SDK), manipulates structured weight domain sparsity to generate a sparse set of static weights for the NN. The structured sparsity static weights are loaded into the NN after compilation and utilized by both the structured weight domain sparsity mechanism and the structured activation domain sparsity mechanism. The application of structured sparsity lowers the span of search options and creates a relatively loose coupling between the data and control planes.
1. A method of weight domain sparsity for use in an artificial neural network (ANN) compiler, the method comprising:
searching a plurality of weight tensors within a weight space for one or more sparsity patterns within a predefined codebook of valid sparsity patterns;
packing a weight memory with packed weight tensors to reduce memory usage in accordance with one or more found sparsity patterns; and
generating one or more weight sparsity instructions corresponding to a skip sequence based on said found sparsity patterns for use in subsequent retrieval of weights and input data from said weight memory and input memory, respectively, during an inference mode of operation.
2. The method according to claim 1 , wherein said packed weight tensors each are configured to represent one or more predetermined weight sparsity patterns that effectively reduce memory usage and power consumption.
3. The method according to claim 1 , wherein said weight sparsity patterns are known a priori and are selected from a group consisting of a vertical column, a horizontal row, a diagonal, an ‘X’ shape, a ‘+’ shape, a triangular block, a single weight, and any combination or superposition thereof.
4. The method according to claim 1 , wherein said one or more weight sparsity instructions are adapted to be subsequently stored in a neural network processor as one or more microcode instructions.
5. The method according to claim 4 , wherein each microcode instruction comprises a plurality of opcodes operative to generate a synchronized sequence of operations on said neural network processor including weights and input data.
6. The method according to claim 1 , wherein said one or more found sparsity patterns are known a priori and stored in one or more configuration registers.
7. A method of weight domain sparsity for use in an artificial neural network (ANN) compiler, the method comprising:
searching a plurality of weights stored in a weight memory for one or more sparsity patterns known a priori;
generating scoring for said plurality of weights for a minimum possible memory size;
reordering said plurality of weights in said weight memory as one or more weight tensors in accordance with corresponding scores;
repeating said steps of searching, scoring, and reordering to reduce weight memory usage in accordance with one or more found patterns;
maximally packing said weight tensors in said weight memory in accordance with one or more found sparsity patterns; and
generating one or more weight sparsity instructions based on said one or more found sparsity patterns for use in subsequent retrieval of weights and input data from said weight memory and input memory, respectively.
8. The method according to claim 7 , wherein said packed weight tensors each are configured to represent one or more predetermined weight sparsity patterns that effectively reduce memory usage and power consumption.
9. The method according to claim 7 , wherein said weight sparsity patterns are known a priori and are selected from a group consisting a vertical column, a horizontal row, a diagonal, an ‘X’ shape, a ‘+’ shape, a triangular block, a single weight, and any combination or superposition thereof.
10. The method of claim 9 wherein said weight sparsity patterns comprise an argument operative to shift weight data vertically, horizontally, and/or to shorten or lengthen one of said weight sparsity patterns.
11. The method according to claim 7 , wherein reordering said plurality of weights comprises rearranging said tensor dimensions.
12. The method according to claim 7 , wherein reordering of said plurality of weights comprises applying a transpose operation.
13. The method according to claim 7 , wherein reordering of said plurality of weights comprises swapping a plurality of axes.
14. The method according to claim 7 , wherein reordering of said plurality of weight comprises unrolling one of said weights into a vector using a row-major order.
15. The method according to claim 7 , wherein reordering of said plurality of weight comprises flipping one or more input data memory locations.
16. The method according to claim 7 , wherein said packed weight memory comprises one or more weight memory tensors representing a predetermined plurality of weight sparsity patterns that effectively reduce memory usage and power consumption.
17. The method according to claim 7 , wherein said one or more weight sparsity instructions are implemented in hardwired circuitry.
18. The method according to claim 7 , wherein said one or more weight sparsity instructions are adapted to be subsequently stored in a neural network processor as one or more microcode instructions.
19. The method according to claim 18 , wherein said microcode instructions comprise a plurality of opcodes operative to generate and effect, on said neural network processor, said subsequent retrieval of weights and input data synchronized appropriately taking into account said one or more memory address skipping operations.