METHODS AND DEVICES FOR NEURAL NETWORK QUANTIZATION USING TEMPORAL PROFILING
Methods and apparatuses are provided for temporal profiling for neural network quantization. The method includes: obtaining a neural network that comprises anode connected to different paths at different time periods; obtaining node outputs for the node at the different time periods; determining statistic properties of the node outputs at the different time periods; and determining activation ranges of the node outputs based on the statistic properties.
1 . A method for neural network quantization, comprising:
obtaining a neural network that comprises a node connected to different paths at different time periods;
obtaining node outputs for the node at the different time periods;
determining statistic properties of the node outputs at the different time periods; and
determining activation ranges of the node outputs based on the statistic properties.
2 . The method of claim 1 , further comprising:
determining additional activation ranges for remaining nodes in the neural network; and
quantizing the neural network by quantizing each layer in the neural network and respectively quantizing each node output based on respective activation range.
3 . The method of claim 2 , wherein the neural network is a recurrent neural network for automatic speech recognition; and
wherein the method further comprises: implementing the recurrent neural network in an edge computing device after quantizing all neural network layers in the recurrent neural network.
4 . The method of claim 1 , wherein the neural network is one of following neural networks: a Long Short-Term Memory (LSTM), a LSTM with recurrent project layer (LSTMP), or a Gated Recurrent Unit (GRU).
5 . The method of claim 1 , wherein obtaining node outputs for the node at the different time periods further comprises:
multiplying input vectors of the node at the different time periods with weight matrices to obtain weighted matrices; and
concatenating the weighted matrices for further processing to obtain the node outputs.
6 . The method of claim 1 , wherein the statistic properties comprise one or a combination of following properties: a mean estimate, a histogram, a probability density function, a variance estimate, an entropy, a cross entropy, or a Kullback-Leiber Divergence.
7 . The method of claim 1 , wherein the neural network is a recurrent neural network for video recognition.
8 . An apparatus for implementing a neural network, comprising:
one or more processors; and
a memory configured to store instructions executable by the one or more processors;
wherein the one or more processors, upon execution of the instructions, are configured to:
obtain a neural network that comprises a node connected to different paths at different time periods;
obtain node outputs for the node at the different time periods;
determine statistic properties of the node outputs at the different time periods; and
determine activation ranges of the node outputs based on the statistic properties.
9 . The apparatus of claim 8 , wherein the one or more processors are further configured to:
determine additional activation ranges for remaining nodes in the neural network; and
quantize the neural network by quantizing each layer in the neural network and respectively quantizing each node output based on respective activation range.
10 . The apparatus of claim 9 , wherein the neural network is a recurrent neural network for automatic speech recognition; and
wherein the one or more processors are further configured to:
implement the recurrent neural network in an edge computing device after quantizing all neural network layers in the recurrent neural network.
11 . The apparatus of claim 8 , wherein the neural network is one of following neural networks: a Long Short-Term Memory (LSTM), a LSTM with recurrent project layer (LSTMP), or a Gated Recurrent Unit (GRU).
12 . The apparatus of claim 8 , wherein the one or more processors are further configured to:
multiply input vectors of the node at the different time periods with weight matrices to obtain weighted matrices; and
concatenate the weighted matrices for further processing to obtain the node outputs.
13 . The apparatus of claim 8 , wherein the statistic properties comprise one or a combination of following properties: a mean estimate, a histogram, a probability density function, a variance estimate, an entropy, a cross entropy or a Kullback-Leiber Divergence.
14 . The apparatus of claim 8 , wherein the neural network is a recurrent neural network for video recognition.
15 . A non-transitory computer readable storage medium, comprising instructions stored therein to implement a neural network, wherein, upon execution of the instructions by one or more processors, the instructions cause the one or more processors to perform acts comprising:
obtaining a neural network that comprises a node connected to different paths at different time periods;
obtaining node outputs for the node at the different time periods;
determining statistic properties of the node outputs at the different time periods; and
determining activation ranges of the node outputs based on the statistic properties.
16 . The non-transitory computer readable storage medium of claim 15 , wherein the instructions cause the one or more processors to further perform:
determining additional activation ranges for remaining nodes in the neural network; and
quantizing the neural network by quantizing each layer in the neural network and respectively quantizing each node output based on respective activation range.
17 . The non-transitory computer readable storage medium of claim 16 , wherein the neural network is a recurrent neural network for automatic speech recognition; and
wherein the instructions cause the one or more processors to further perform:
implementing the recurrent neural network in an edge computing device after quantizing all neural network layers in the recurrent neural network.
18 . The non-transitory computer readable storage medium of claim 15 , wherein the neural network is one of following neural networks: a Long Short-Term Memory (LSTM), a LSTM with recurrent project layer (LSTMP), or a Gated Recurrent Unit (GRU).
19 . The non-transitory computer readable storage medium of claim 15 , wherein obtaining node outputs for the node at the different time periods further comprises:
multiplying input vectors of the node at the different time periods with weight matrices to obtain weighted matrices; and
concatenating the weighted matrices for further processing to obtain the node outputs.
20 . The non-transitory computer readable storage medium of claim 15 , wherein the statistic properties comprise one or a combination of following properties: a mean estimate, a histogram, a probability density function, a variance estimate, an entropy, a cross entropy or a Kullback-Leiber Divergence.