Quantization of neural network parameters
A method of generating a neural network may be provided. A method may include applying non-linear quantization to a plurality of synaptic weights of a neural network model. The method may further include training the neural network model. Further, the method ma include generating a neural network output from the trained neural network model based on or more inputs received by the trained neural network model.
1. A method of operating a neural network, comprising
applying, via at least one processor, non-linear quantization to a plurality of synaptic weights of a neural network model, wherein applying the non-Ii near quantization comprises:
quantizing a first synaptic weight of the plurality of synaptic weights with a coarse resolution of N number bits in response to the first synaptic weight having a first value; and
quantizing a second synaptic weight of the plurality of synaptic weights with a fine resolution of M number bits in response to the second synaptic weight having a second value, wherein M is greater than N and the second synaptic weight is quantized with M number of hits instead of N number of bits in response to the second value being greater than the first value;
training, via the at least one processor, the neural network model; and
generating, via the at least one processor, a neural network output from the trained neural network model based on one or more inputs received by the trained neural network model.
2. The method of claim 1 , further comprising determining an accuracy of the neural network model, wherein the generating the neural network output comprises generating, in response to the determined accuracy being greater than a threshold accuracy, the neural network output.
3. The method of claim 1 , wherein applying the non-linear quantization comprises applying a non-linear function to each synaptic weight of the plurality of synaptic weights of the neural network model.
4. The method of claim 3 , wherein applying the non-linear function comprises applying one of a hyperbolic sine function and a hyperbolic cosine function to each synaptic weight of the plurality of synaptic weights.
5. The method of claim 1 , further comprising:
determining an accuracy of the neural network model; and
increasing resolution of the neural network via increasing a number of bits of the non-linear quantization with respect to one or more of the synaptic weights in response to the determined accuracy being less than a threshold accuracy.
6. The method of claim 1 , wherein applying the non-linear quantization comprises applying a non-linear function to the plurality of synaptic weights of the neural network model to generate non-uniformly distributed quantization levels.
7. The method of claim 1 , wherein applying the non-linear quantization comprises applying a non-linear function to the plurality of synaptic weights of the neural network model to generate uniformly distributed quantization levels.
8. One or more non-transitory computer-readable media that include instructions that, when executed by one or more processors, are configured to cause the one or more processors to perform operations, the operations comprising:
applying non-linear quantization to a plurality of synaptic weights of a neural network model, wherein applying the non-linear quantization comprises:
quantizing a first synaptic weight of the plurality of synaptic weights with a coarse resolution of N number bits in response to the first synaptic weight having a first value; and
quantizing a second synaptic weight of the plurality of synaptic weights with a fine resolution of M number bits in response to the second synaptic weight having a second value, wherein M is greater than N and the second synaptic weight is quantized with M number of bits instead of N number of bits in response to the second value being greater than the first value;
training the neural network model; and
generating a neural network output from the trained neural network model based on one or more inputs received by the trained neural network model.
9. The computer-readable media of claim 8 , the operations further comprising determining an accuracy of the neural network model, wherein the generating the neural network output comprises generating, in response to the determined accuracy being greater than a threshold accuracy, the neural network output.
10. The computer-readable media of claim 8 , wherein applying the non-linear quantization comprises applying a non-linear function to each synaptic weight of the plurality of synaptic weights of the neural network model.
11. The computer-readable media of claim 10 , wherein applying the non-linear function comprises applying one of a hyperbolic sine function and a hyperbolic cosine function to each synaptic weight of the plurality of synaptic weights.
12. The computer-readable media of claim 8 , the operations further comprising:
determining an accuracy of the neural network model; and
increasing resolution of the neural network via increasing a number of bits of the non-linear quantization with respect to one or more of the synaptic weights in response to the determined accuracy being less than a threshold accuracy.
13. The computer-readable media of claim 8 , wherein applying the non-linear quantization comprises applying a non-linear function to the plurality of synaptic weights of the neural network model to generate non-uniformly distributed quantization levels.
14. The computer-readable media of claim 8 , wherein applying the non-linear quantization comprises applying a non-linear function to the plurality of synaptic weights of the neural network model to generate uniformly distributed quantization levels.