Optimized quantization for reduced resolution neural networks

1. A machine learning system comprising:

at least one controller including a hardware processor, wherein the at least one controller is programmed to convert floating-point weighting factors for a neural network into fixed-point weighting factors using a scaling factor that is a multiple of a predetermined base, b, and that minimizes a cost function that is a mean square error between the floating-point weighting factors and a product of candidate scaling factors and corresponding fixed-point weighting factors, and, after a predetermined number of iterations during a training stage, change the scaling factor.

2. The machine learning system of claim 1 , wherein the controller is further programmed to implement the neural network using fixed-point operations.

3. The machine learning system of claim 1 , wherein the candidate scaling factors include first and second candidate values having exponents L and L−1, respectively, such that a mean of absolute values of the floating-point weighting factors is between b L and b L−1 .

4. The machine learning system of claim 3 , wherein the controller is further programmed to evaluate the cost function with candidate scaling factors that are multiples of the predetermined base from b L−1 to b L+4 .

5. The machine learning system of claim 1 , wherein the controller is further programmed to evaluate the cost function for a first number of candidate scaling factors that are greater than a mean of absolute values of the floating-point weighting factors and for a second number of candidate scaling factors that are less than the mean, the first number being greater than the second number.

6. The machine learning system of claim 1 , wherein the controller is further programmed to, after completion of the training stage, provide the fixed-point weighting factors to an inference stage that is configured to implement a neural network.

7. The machine learning system of claim 1 , wherein the predetermined base, b, is two.

8. The machine learning system of claim 1 , wherein the controller is further programmed to define the scaling factor for a layer that includes more than one node.

9. A method comprising:

selecting a predetermined number of candidate scaling factors that are multiples of two;

evaluating, for each of the candidate scaling factors, a cost function that is a mean square error between a predetermined set of floating-point weighting factors for a neural network and a product of a candidate scaling factors being evaluated and fixed-point weighting factors defined by the candidate scaling factor being evaluated;

selecting a scaling factor as a one of the candidate scaling factors that results in a minimum value of the cost function;

generating a set of fixed-point weighting factors by scaling each of the floating-point weighting factors by the scaling factor; and

implementing the neural network using the set of fixed-point weighting factors.

10. The method of claim 9 , wherein the candidate scaling factors include first and second candidate values having exponents L and L−1, respectively, such that a mean of absolute values of the predetermined set of floating-point weighting factors is between 2 L and 2 L−1 .

11. The method of claim 10 , wherein the candidate scaling factors include multiples of two from 2 L−1 to 2 L+4 .

12. The method of claim 9 , wherein the candidate scaling factors include a greater number of candidate scaling factors that are greater than a mean of absolute values of the floating-point weighting factors than are less than the mean.

13. The method of claim 9 , wherein the predetermined set corresponds to a node of the neural network.

14. A method for converting floating-point weighting factors to fixed-point weighting factors for a neural network, the method comprising:

selecting a predetermined number of candidate scaling factors that are multiples of a predetermined base, wherein the predetermined number of candidate scaling factors includes a greater number of candidates having a value that exceeds a mean value of an absolute value of the floating-point weighting factors;

evaluating each of the candidate scaling factors in a cost function;

selecting a scaling factor as a one of the candidate scaling factors that results in a minimum value of the cost function;

generating the fixed-point weighting factors by scaling the floating-point weighting factor using the scaling factor; and

implementing the neural network using the fixed-point weighting factors.

15. A method for converting floating-point weighting factors to fixed-point weighting factors for a neural network, the method comprising:

selecting a predetermined number of candidate scaling factors that are multiples of a predetermined base, wherein the predetermined number of candidate scaling factors includes only one candidate that is less than a mean value of absolute values of the floating-point weighting factors associated;

evaluating each of the candidate scaling factors in a cost function;

selecting a scaling factor as a one of the candidate scaling factors that results in a minimum value of the cost function;

generating the fixed-point weighting factors by scaling the floating-point weighting factor using the scaling factor; and

implementing the neural network using the fixed-point weighting factors.

16. A method for converting floating-point weighting factors to fixed-point weighting factors for a neural network, the method comprising:

selecting a predetermined number of candidate scaling factors that are multiples of a predetermined base;

evaluating each of the candidate scaling factors in a cost function;

selecting a scaling factor as a one of the candidate scaling factors that results in a minimum value of the cost function;

generating the fixed-point weighting factors by scaling the floating-point weighting factor using the scaling factor, wherein the cost function is a mean square error between the floating-point weighting factors and a product of the candidate scaling factors and corresponding fixed-point weighting factors; and

implementing the neural network using the fixed-point weighting factors.

17. The method of claim 16 , wherein the predetermined base is two.

18. The method of claim 16 , further comprising:

responsive to completing a training stage for the neural network, providing the fixed-point weighting factors to an inference stage.

19. The method of claim 16 , further comprising:

updating the scaling factor during a training stage of a neural network after a predetermined number of training intervals.