AUTOMATIC QUANTIZATION OF A FLOATING POINT MODEL

1 . A system comprising:

a processor; and

a memory storing instructions that, when executed by the processor, configure the processor to perform operations comprising:

providing a floating point model to an automatic quantization library, the floating point model being configured to represent a neural network, and the automatic quantization library being configured to generate a first quantized model based on the floating point model;

providing a function to the automatic quantization library, the function being configured to run a forward pass on a given dataset for the floating point model;

causing the automatic quantization library to generate the first quantized model based on the floating point model;

causing the automatic quantization library to calibrate the first quantized model by running the first quantized model on the function; and

converting the calibrated first quantized model to a second quantized model.

2 . The system of claim 1 , the operations further comprising:

providing the second quantized model to a client device, wherein the second quantized model is configured to represent the neural network and to be accessible by an application running on the client device.

3 . The system of claim 2 , wherein the automatic quantization library is usable by developers to quantize floating point models representing respective neural networks, for deployment to client devices running the application.

4 . The system of claim 1 , the operations further comprising, prior to converting the calibrated first quantized model to the second quantized model:

exporting the calibrated first quantized model to an open neural network exchange format,

wherein converting the calibrated first quantized model to the second quantized model is based on the open neural network exchange format.

5 . The system of claim 1 , wherein each of the floating point model and the first quantized model corresponds to a 32-bit floating point model.

6 . The system of claim 5 , wherein the second quantized model corresponds to an 8-bit quantized model.

7 . The system of claim 5 , wherein the calibrated first quantized model corresponds to a 32-bit floating point model with quantization statistics.

8 . The system of claim 1 , wherein the automatic quantization library is configured to:

generate a computational graph based on the floating point model;

for each node in the computational graph,

extract a layer type for the node,

match the layer type to a quantized layer type supported by the automatic quantization library, and

replace the node with a quantized node corresponding to the quantized layer type; and

provide the first quantized model as output based on replacing the nodes with the quantized nodes.

9 . A method, comprising:

providing a floating point model to an automatic quantization library, the floating point model being configured to represent a neural network, and the automatic quantization library being configured to generate a first quantized model based on the floating point model;

providing a function to the automatic quantization library, the function being configured to run a forward pass on a given dataset for the floating point model;

causing the automatic quantization library to generate the first quantized model based on the floating point model;

causing the automatic quantization library to calibrate the first quantized model by running the first quantized model on the function; and

converting the calibrated first quantized model to a second quantized model.

10 . The method of claim 9 , further comprising:

providing the second quantized model to a client device, wherein the second quantized model is configured to represent the neural network and to be accessible by an application running on the client device.

11 . The method of claim 10 , wherein the automatic quantization library is usable by developers to quantize floating point models representing respective neural networks, for deployment to client devices running the application.

12 . The method of claim 9 , further comprising, prior to converting the calibrated first quantized model to the second quantized model:

exporting the calibrated first quantized model to an open neural network exchange format,

wherein converting the calibrated first quantized model to the second quantized model is based on the open neural network exchange format.

13 . The method of claim 9 , wherein each of the floating point model and the first quantized model corresponds to a 32-bit floating point model.

14 . The method of claim 13 , wherein the second quantized model corresponds to an 8-bit quantized model.

15 . The method of claim 13 , wherein the calibrated first quantized model corresponds to a 32-bit floating point model with quantization statistics.

16 . The method of claim 9 , wherein the automatic quantization library is configured to:

generate a computational graph based on the floating point model;

for each node in the computational graph,

extract a layer type for the node,

match the layer type to a quantized layer type supported by the automatic quantization library, and

replace the node with a quantized node corresponding to the quantized layer type; and

provide the first quantized model as output based on replacing the nodes with the quantized nodes.

17 . A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising:

providing a floating point model to an automatic quantization library, the floating point model being configured to represent a neural network, and the automatic quantization library being configured to generate a first quantized model based on the floating point model;

providing a function to the automatic quantization library, the function being configured to run a forward pass on a given dataset for the floating point model;

causing the automatic quantization library to generate the first quantized model based on the floating point model;

causing the automatic quantization library to calibrate the first quantized model by running the first quantized model on the function; and

converting the calibrated first quantized model to a second quantized model.

18 . The non-transitory computer-readable storage medium of claim 17 , the operations further comprising:

providing the second quantized model to a client device, wherein the second quantized model is configured to represent the neural network and to be accessible by an application running on the client device.

19 . The non-transitory computer-readable storage medium of claim 18 , wherein the automatic quantization library is usable by developers to quantize floating point models representing respective neural networks, for deployment to client devices running the application.

20 . The non-transitory computer-readable storage medium of claim 17 , the operations further comprising, prior to converting the calibrated first quantized model to the second quantized model:

exporting the calibrated first quantized model to an open neural network exchange format,

wherein converting the calibrated first quantized model to the second quantized model is based on the open neural network exchange format.