Implementing neural networks in fixed point arithmetic computing systems

1. A method comprising:

receiving a request to process a neural network using a processing system that performs neural network computations using fixed point arithmetic,

the neural network comprising a plurality of layers including an input layer,

each layer having a plurality of nodes, and

each node of each layer having a set of floating point weight values;

for each node of each layer, determining a respective scaling value for the node from the respective set of floating point weight values for the node;

receiving a set of floating point input values for each of the plurality of nodes of the input layer of the neural network;

for each of the plurality of nodes of the input layer, converting each floating point input value of the set of floating point input values for the node to a corresponding fixed point value using the respective scaling value for the node to generate a set of fixed point input values for the node; and

providing each fixed point floating point input value to the processing system for use in processing inputs using the neural network.

2. The method of claim 1 , wherein converting each floating point input value for a node of the input layer into a corresponding fixed point value comprises dividing the floating point input value by the respective scaling value for the node.

3. The method of claim 1 , further comprising:

receiving a set of fixed point output values generated by the processing system for a last layer of the plurality of layers, each fixed point output value corresponding to a respective node of the last layer; and

converting each of the fixed point output values to a respective floating point output value using the scaling value for each node of the last layer corresponding to the fixed point output value.

4. The method of claim 3 , wherein converting each fixed point output value to a corresponding floating point output value comprises dividing the fixed point output value by the respective scaling value for the node having the fixed point output value.

5. The method of claim 1 , further comprising:

for each node of each layer, converting each floating point weight value of the node into a corresponding fixed point weight value using the respective scaling value for the node to generate a set of fixed point weight values for the node.

6. The method of claim 5 , wherein converting each floating point weight value to a corresponding fixed-point weight value comprises multiplying each floating point weight value by the respective scaling value for the node having the floating point weight value.

7. The method of claim 1 , wherein the respective set of floating point weight values associated with each node of each layer represents a respective row or respective column in a weight matrix associated with the layer.

8. The method of claim 1 , wherein determining the respective scaling value for each respective node is performed from the set of floating point weight values associated with the respective node and one or more sets of floating point weight values each associated with a node other than the respective node.

9. The method of claim 1 , wherein determining the respective scaling value for the node comprises:

identifying, for each set of floating point weight values associated with the node, the floating point weight value with the greatest magnitude;

identifying a largest representable value in the fixed point arithmetic used by the processing system to perform neural network computations;

computing the respective scaling value for each set of floating point weight values based on a ratio of the magnitude of the respective floating point weight value with the greatest magnitude and the maximum representable value in the fixed point arithmetic.

10. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

receiving a request to process a neural network using a processing system that performs neural network computations using fixed point arithmetic,

the neural network comprising a plurality of layers including an input layer,

each layer having a plurality of nodes, and

each node of each layer having a set of floating point weight values;

for each node of each layer, determining a respective scaling value for the node from the respective set of floating point weight values for the node;

receiving a set of floating point input values for each of the plurality of nodes of the input layer of the neural network;

for each of the plurality of nodes of the input layer, converting each floating point input value of the set of floating point input values for the node to a corresponding fixed point value using the respective scaling value for the node to generate a set of fixed point input values for the node; and

providing each fixed point floating point input value to the processing system for use in processing inputs using the neural network.

11. The system of claim 10 , wherein converting each floating point input value for a node of the input layer into a corresponding fixed point value comprises dividing the floating point input value by the respective scaling value for the node.

12. The system of claim 10 , further comprising:

receiving a set of fixed point output values generated by the processing system for a last layer of the plurality of layers, each fixed point output value corresponding to a respective node of the last layer; and

converting each of the fixed point output values to a respective floating point output value using the scaling value for each node of the last layer corresponding to the fixed point output value.

13. The system of claim 12 , wherein converting each fixed point output value to a corresponding floating point output value comprises dividing the fixed point output value by the respective scaling value for the node having the fixed point output value.

14. The system of claim 10 , further comprising:

for each node of each layer, converting each floating point weight value of the node into a corresponding fixed point weight value using the respective scaling value for the node to generate a set of fixed point weight values for the node.

15. The system of claim 14 , wherein converting each floating point weight value to a corresponding fixed-point weight value comprises multiplying each floating point weight value by the respective scaling value for the node having the floating point weight value.

16. The system of claim 10 , wherein the respective set of floating point weight values associated with each node of each layer represents a respective row or respective column in a weight matrix associated with the layer.

17. The system of claim 10 , wherein determining the respective scaling value for each respective node is performed from the set of floating point weight values associated with the respective node and one or more sets of floating point weight values each associated with a node other than the respective node.

18. The system of claim 10 , wherein determining the respective scaling value for the node comprises:

identifying, for each set of floating point weight values associated with the node, the floating point weight value with the greatest magnitude;

identifying a largest representable value in the fixed point arithmetic used by the processing system to perform neural network computations;

computing the respective scaling value for each set of floating point weight values based on a ratio of the magnitude of the respective floating point weight value with the greatest magnitude and the maximum representable value in the fixed point arithmetic.

19. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

receiving a request to process a neural network using a processing system that performs neural network computations using fixed point arithmetic,

the neural network comprising a plurality of layers including an input layer,

each layer having a plurality of nodes, and

each node of each layer having a set of floating point weight values;

for each node of each layer, determining a respective scaling value for the node from the respective set of floating point weight values for the node;

receiving a set of floating point input values for each of the plurality of nodes of the input layer of the neural network;

for each of the plurality of nodes of the input layer, converting each floating point input value of the set of floating point input values for the node to a corresponding fixed point value using the respective scaling value for the node to generate a set of fixed point input values for the node; and

providing each fixed point floating point input value to the processing system for use in processing inputs using the neural network.

20. The computer storage medium of claim 19 , further comprising:

receiving a set of fixed point output values generated by the processing system for a last layer of the plurality of layers, each fixed point output value corresponding to a respective node of the last layer; and

converting each of the fixed point output values to a respective floating point output value using the scaling value for each node of the last layer corresponding to the fixed point output value.