Apparatus and methods for neural network operations supporting fixed point numbers of short bit length

1. An apparatus for neural network operations, comprising:

a fixed-point number converter configured to convert one or more first floating-point numbers to one or more first fixed-point numbers in accordance with at least one format;

a neural network processor configured to process the first fixed-point numbers to generate one or more process results, and

a floating-point number analyzing processor configured to determine the at least one format of fixed-point numbers, wherein the floating-point number analyzing processor includes:

a data extractor configured to collect one or more categories of the first floating-point numbers;

a data analyzer configured to statistically analyze the one or more categories of the first floating-point numbers to determine a distribution pattern for each of the one or more categories over one or more data ranges; and

a format determiner configured to determine the at least one format of fixed-point numbers for the one or more categories.

2. The apparatus of claim 1 , wherein the format determiner is configured to determine one of the at least one format of fixed-point numbers for each of the one or more categories.

3. The apparatus of claim 1 , wherein each of the at least one format includes a point location that indicates a location of the binary point.

4. The apparatus of claim 1 , further comprising a data cache configured to store the one or more process results.

5. The apparatus of claim 1 , wherein the fixed-point number converter includes:

a pruning processor configured to adjust a length of a mantissa field of the first floating-point numbers; and

a point controller configured to set a binary point to the first floating-point numbers in accordance with the at least one format.

6. The apparatus of claim 5 , wherein the pruning processor includes a random trimmer configured to

adjust the length of the mantissa field of the first floating-point numbers to a first length in accordance with a first probability; and

adjust the length of the mantissa field of the first floating-point numbers to a second length in accordance with a second probability.

7. The apparatus of claim 5 , wherein the pruning processor includes a half-adjust trimmer configured to:

add an integer value to the mantissa field if a fraction indicated by the mantissa field is not less than one-half of a smallest positive integer representable by the format of fixed-point numbers, wherein the integer value is equal to the smallest positive integer representable by the format of fixed-point numbers; and

clear the fraction indicated by the mantissa field if the fraction is less than one-half of the smallest positive integer representable by the format of fixed-point numbers.

8. The apparatus of claim 5 , wherein the pruning processor includes a round-up trimmer configured to round up the mantissa field to a smallest positive integer that is greater than the first floating-point number.

9. The apparatus of claim 5 , wherein the pruning processor includes a round-down trimmer configured to round down the mantissa field to a greatest positive integer that is less than the first floating-point number.

10. The apparatus of claim 5 , wherein the pruning processor includes a cut-off trimmer configured to discard mantissa digits that exceed a length of a mantissa field in accordance with the at least one format of fixed-point numbers.

11. The apparatus of claim 1 , further comprising:

a SL floating-point number converter configured to convert the one or more process results to one or more second floating-point numbers;

a backpropagation module configured to process the one or more second floating-point numbers to generate one or more backpropagation results; and

a LS floating-point number converter configured to convert the one or more backpropagation results to one or more second fixed-point numbers.

12. A method for neural network operations, comprising:

converting, by a fixed-point number converter, one or more first floating-point numbers to one or more first fixed-point numbers in accordance with at least one format;

processing, by a neural network processor, the first fixed-point numbers to generate one or more process results, and

collecting, by a data extractor of a floating-point number analyzing processor, one or more categories of the first floating-point numbers;

statistically analyzing, by a data analyzer of a floating-point number analyzing processor, the one or more categories of the first floating-point numbers to determine a distribution pattern for each of the one or more categories over one or more data ranges; and

determining, by a format determiner of a floating-point number analyzing processor, the at least one format of fixed-point numbers for the one or more categories.

13. The method of claim 12 , further comprising determining, by the format determiner, one of the at least one format of fixed-point numbers for each of the one or more categories.

14. The method of claim 12 , wherein each of the at least one format includes a point location that indicates a location of the binary point.

15. The method of claim 12 , further comprising

adjusting, by a pruning processor of the fixed-point number converter, a length of a mantissa field of the first floating-point numbers; and

setting, by a point controller of the fixed-point number converter, a binary point to the first floating-point numbers in accordance with the at least one format.

16. The method of claim 15 , wherein the adjusting the length of the mantissa field further comprises:

adjusting, by a random trimmer of the pruning processor, the length of the mantissa field of the first floating-point numbers to a first length in accordance with a first probability; and

adjusting, by the random trimmer of the pruning processor, the length of the mantissa field of the first floating-point numbers to a second length in accordance with a second probability.

17. The method of claim 15 , wherein the adjusting the length of the mantissa field further comprises:

adding, by a half-adjust trimmer of the pruning processor, one to a second least significant bit of the mantissa field if a least significant digit is not less than one-half of a smallest positive integer representable by the format of fixed-point numbers; and

clearing, by the half-adjust trimmer of the pruning processor, the least significant digit if the least significant digit is less than one-half of a smallest positive integer representable by the format of fixed-point numbers.

18. The method of claim 15 , wherein the adjusting the length of the mantissa field further comprises rounding up, by a round-up trimmer of the pruning processor, the mantissa field to a smallest positive integer that is greater than the first floating-point number.

19. The method of claim 15 , wherein the adjusting the length of the mantissa field further comprises rounding down, by a round-down trimmer, the mantissa field to a greatest positive integer that is less than the first floating-point number.

20. The method of claim 15 , wherein the adjusting the length of the mantissa field further comprises discarding, by a cut-off trimmer, exponent digits that exceed a length of the mantissa field in accordance with the at least one format of fixed-point numbers.

21. The method of claim 12 , further comprising storing, by a data cache, the one or more process results.

22. The method of claim 12 , further comprising:

converting, by a SL floating-point number converter, the one or more process results to one or more second floating-point numbers;

processing, by a backpropagation module, the one or more second floating-point numbers to generate one or more backpropagation results; and

converting, by a LS floating-point number converter, the one or more backpropagation results to one or more second fixed-point numbers.