Fixed-point training method for deep neural networks based on dynamic fixed-point conversion scheme
The present disclosure proposes a fixed-point training method and apparatus based on dynamic fixed-point conversion scheme. More specifically, the present disclosure proposes a fixed-point training method for LSTM neural network. According to this method, during the fine-tuning process of the neural network, it uses fixed-point numbers to conduct forward calculation. Accordingly, within several training cycles, the network accuracy may returned to the desired accuracy level under floating point calculation.
1. A fixed-point training method for a floating point neural network, comprising
a fixed-point neural network obtaining step, for converting weights in said floating point neural network to fixed-point numbers, so as to obtain a corresponding fixed-point neural network;
a fixed-point conversion scheme determining step, comprising:
conducting a floating point forward calculation of said fixed-point neural network using a current data input in a training dataset, so as to obtain a current temporary fixed-point conversion scheme;
comparing said current temporary fixed-point conversion scheme with a previous fixed-point conversion scheme, said previous fixed-point conversion scheme corresponding to a previous data input, and selecting an optimal fixed-point conversion scheme as an updated current fixed-point conversion scheme corresponding to the current data input such that said updated current fixed-point conversion scheme reduces a number of overflow instances in a fixed-point forward calculation of said fixed-point neural network using said current data input based on said updated current fixed-point conversion scheme;
a fixed-point forward calculation step, for conducting a fixed-point forward calculation of said fixed-point neural network using said current data input based on said updated current fixed-point conversion scheme; and
a neural network updating step, for updating said floating point neural network using the calculation result of said fixed-point forward calculation.
2. The method according to claim 1 , further comprises: executing iteratively said fixed-point neural network obtaining step, said fixed-point conversion scheme determining step, said fixed-point forward calculation step and said neural network updating step, until the floating point neural network achieves a desired accuracy.
3. The method according to claim 1 , wherein said fixed-point conversion scheme at least includes: the total bit length and the decimal bit length of a fixed-point number used for representing a corresponding floating point number.
4. The method according to claim 1 , wherein said fixed-point neural network obtaining step further comprises: converting weights of said floating point neural network to fixed-point numbers by a rule of fixed-point conversion of floating point number.
5. The method according to claim 1 , wherein in said fixed-point conversion scheme determining step, the fixed-point conversion scheme with a smaller decimal bit length is selected as the optimal scheme.
6. The method according to claim 1 , wherein said fixed-point forward calculation step further comprises: converting the input data used for forward calculation and intermediate calculation results of said forward calculation to fixed-point numbers on the basis of said updated current fixed-point conversion scheme.
7. The method according to claim 1 , wherein said floating point neural network updating step further comprises:
comparing calculation result of said fixed-point forward calculation with a standard value to obtain an accuracy loss; and
calculating a floating point gradient based on said accuracy loss, and updating said floating point neural network based on said floating point gradient.
8. The method according to claim 1 , wherein said method is used for conducting a fixed-point fine-tuning for a trained floating point neural network.
9. The method according to claim 1 , wherein said method is used for conducting a fixed-point training for a raw floating point neural network.
10. The method according to claim 1 , wherein said floating point neural network is an LSTM neural network.
11. The method according to claim 1 , wherein said training dataset is a voice dataset.
12. A fixed-point training method for a floating point neural network, comprising
a fixed-point neural network obtaining step, for converting weights of said floating point neural network to fixed-point numbers, so as to obtain a fixed-point neural network;
a first forward calculation step, for conducting a floating point forward calculation of said fixed-point neural network using a current data input of a training database, so as to obtain the value range of intermediate calculation results and generate a current temporary fixed-point conversion scheme;
a fixed-point conversion scheme determining step, for comparing the current temporary fixed-point conversion scheme with a previous fixed-point conversion scheme, said previous fixed-point conversion scheme being determined by a previous data input of the training database, and selecting the fixed-point conversion scheme with a smaller decimal length as an updated current fixed-point conversion scheme such that said updated current fixed-point conversion scheme reduces a number of overflow instances in a fixed-point forward calculation of said fixed-point neural network using said current data input based on said updated current fixed-point conversion scheme;
a second forward calculation step, for conducting a fixed-point forward calculation of said fixed-point neural network using said current data input based on said updated current fixed-point conversion scheme;
a neural network updating step, for calculating a floating point gradient using the calculation result obtained in said second forward calculation step, and updating said floating point neural network using said floating point gradient.
13. The method according to claim 12 , further comprises: executing iteratively said fixed-point neural network obtaining step, said first forward calculation step, said fixed-point conversion scheme determining step, said second forward calculation step and said neural network updating step, until the floating point neural network reaches a desired accuracy.
14. A fixed-point training apparatus for a floating point neural network, comprising
a fixed-point neural network obtaining module, for converting weights in said floating point neural network to fixed-point numbers, so as to obtain a corresponding fixed-point neural network;
a fixed-point conversion scheme determining module, comprising:
a current temporary fixed-point conversion scheme determining unit, for conducting a floating point forward calculation of said fixed-point neural network using a current data input in a training dataset, so as to obtain a current temporary fixed-point conversion scheme;
a scheme selecting unit, for comparing said current temporary fixed-point conversion scheme with a previous fixed-point conversion scheme corresponding to a previous data input, and selecting an optimal fixed-point conversion scheme as an updated current fixed-point conversion scheme corresponding to the current data input such that said updated current fixed-point conversion scheme reduces a number of overflow instances in a fixed-point forward calculation of said fixed-point neural network using said current data input based on said updated current fixed-point conversion scheme;
a fixed-point forward calculation module, for conducting a fixed-point forward calculation of said fixed-point network using said current data input based on said updated current fixed-point conversion scheme; and
a neural network updating module, for updating said floating point neural network using the calculation result of said fixed-point forward calculation.
15. The apparatus according to claim 14 , wherein said fixed-point conversion scheme includes: the total bit length and the decimal bit length of a fixed-point number used for representing a corresponding floating point number.
16. The apparatus according to claim 14 , wherein said scheme selecting unit is configured to select the fixed-point conversion scheme with a smaller decimal bit length as the optimal scheme.
17. The apparatus according to claim 14 , wherein said fixed-point neural network obtaining module is configured to convert weights of said floating point neural network to fixed-point numbers by a rule of fixed-point conversion of floating point number.
18. The apparatus according to claim 14 , wherein said fixed-point forward calculation module is configured to convert the input data used for forward calculation and intermediate calculation results of said forward calculation to fixed-point numbers on the basis of said updated current fixed-point conversion scheme.
19. The apparatus according to claim 14 , wherein said neural network updating step further comprises:
a comparing unit, for comparing the calculation result of said fixed-point forward calculation with a standard value to obtain an accuracy loss; and
an updating unit, for calculating a floating point gradient based on said accuracy loss, and updating said floating point neural network based on said floating point gradient.
20. The apparatus according to claim 14 , wherein said apparatus is used for conducting a fixed-point fine-tuning for a trained floating point neural network.
21. The apparatus according to claim 14 , wherein said apparatus is used for conducting a fixed-point training for a raw floating point neural network.