Method for automatic hybrid quantization of deep artificial neural networks

1. A method for quantizing an artificial neural network, the method comprising:

converting a set of floating-point layers in a floating-point network to a set of low-bit-width layers;

for each low-bit-width layer in the set of low-bit-width layers:

calculating a per-layer deviation statistic of the low-bit-width layer based on a mean squared error of a difference between an example output activation of a floating-point layer, in the set of floating-point layers, and a low-bit-width output activation of the low-bit-width layer; and

sorting the low-bit-width layer in the set of low-bit-width layers based on the per-layer deviation statistic as a set of ordered low-bit-width layers;

generating a quantized network representing the floating-point network and comprising the set of low-bit-width layers; and

in response to an accuracy of the quantized network falling below a loss-of-accuracy threshold, sequentially, according to the set of ordered low-bit-width layers:

converting a floating-point layer, represented by a low-bit-width layer in the set of ordered low-bit-width layers, to a high-bit-width layer; and

replacing the low-bit-width layer with the high-bit-width layer in the quantized network.

2. The method of claim 1 :

further comprising:

for each floating-point layer in the set of floating-point layers, calculating a set of example output activations of the floating-point layer based on a set of validation examples; and

for each low-bit-width layer in the set of low-bit-width layers, calculating a set of low-bit-width output activations of the low-bit-width layer based on a set of example input activations; and

wherein calculating the per-layer deviation statistic comprises calculating the per-layer deviation statistic of the low-bit-width layer based on a set of error metrics between the set of low-bit-width output activations of the low-bit-width layer and the set of example output activations of the floating-point layer.

3. The method of claim 2 , further comprising calculating a set of example input activations based on the set of validation examples and a subset of floating-point layers in the set of floating-point layers.

4. The method of claim 1 , wherein sorting the low-bit-width layer in the set of low-bit-width layers comprises sorting the low-bit-width layer in the set of low-bit-width layers, as the set of ordered low-bit-width layers, based on the per-layer deviation statistic and a layer type of the low-bit-width layer.

5. The method of claim 4 , wherein sorting the low-bit-width layer in the set of low-bit-width layers comprises assigning a first low-bit-width layer, exhibiting a first layer type, a first rank in the set of ordered low-bit-width layers exceeding a second rank assigned to a second low-bit-width layer exhibiting a second layer type.

6. The method of claim 5 , wherein sorting the low-bit-width layer in the set of low-bit-width layers comprises assigning the first low-bit-width layer, exhibiting the first layer type representing a convolutional low-bit-width layer, the first rank in the set of ordered low-bit-width layers exceeding the second rank assigned to the second low-bit-width layer exhibiting the second layer type representing a pooling low-bit-width layer.

7. The method of claim 1 :

further comprising:

calculating a first accuracy of the floating-point network; and

for each floating-point layer in the set of floating point layers:

calculating a second accuracy of the floating-point network based on a low-bit-width layer, corresponding to the floating-point layer, and the set of floating point layers excluding the floating point layer; and

calculating a layer priority for the low-bit-width layer corresponding to the floating-point layer based on a difference between the second accuracy and the first accuracy; and

wherein sorting the low-bit-width layer in the set of low-bit-width layers comprises sorting the low-bit-width layer in the set of low-bit-width layers, as the set of ordered low-bit-width layers, based on the per-layer deviation statistic and the layer priority for the low-bit-width layer.

8. The method of claim 1 :

wherein converting the set of floating-point layers comprises converting the set of floating-point layers to a set of eight-bit layers;

wherein generating the quantized network comprises generating the quantized network comprising the set of eight-bit layers;

wherein converting the floating-point layer comprises converting the floating-point layer, represented by an eight-bit layer in a set of ordered eight-bit layers, to a sixteen-bit layer; and

wherein replacing the low-bit-width layer comprises replacing the eight-bit layer with the sixteen-bit layer in the quantized network.

9. The method of claim 1 , further comprising:

recalculating the accuracy of the quantized network as an updated accuracy; and

in response to the updated accuracy exceeding the loss-of-accuracy threshold, loading the quantized network onto an edge device.

10. The method of claim 1 , further comprising:

recalculating the accuracy of the quantized network as an updated accuracy; and

in response to the updated accuracy exceeding the loss-of-accuracy threshold, returning the quantized network.

11. The method of claim 1 , wherein converting the set of floating-point layers comprises converting a floating-point output channel to a low-bit-width output channel in a low-bit-width layer in the set of low-bit-width layers.

12. The method of claim 1 , wherein converting the set of floating-point layers to the set of low-bit-width layers comprises converting a first floating-point layer to a first low-bit-width layer represented in Q-format fixed-point notation specifying:

a first quantity of integer bits; and

a second quantity of fractional bits.

13. The method of claim 12 , wherein converting the first floating-point layer comprises:

converting a set of floating-point weights of the first floating-point layer to a set of Q-format fixed-point weights of the first low-bit-width layer;

converting a bias of the first floating-point layer to a Q-format fixed-point bias of the first low-bit-width layer; and

bit-shifting the set of Q-format fixed-point weights to match the Q-format fixed-point bias.

14. The method of claim 1 :

further comprising calculating a performance metric of the quantized network based on a set of validation examples; and

wherein converting the floating-point layer comprises converting the floating-point layer, represented by the low-bit-width layer in the set of ordered low-bit-width layers, to the high-bit-width layer in response to:

the accuracy of the quantized network falling below the loss-of-accuracy threshold; and

the performance metric of the quantized network exceeding a performance metric threshold.

15. A method for quantizing an artificial neural network, the method comprising:

converting a set of floating-point layers in a floating-point network to a set of low-bit-width layers;

for each low-bit-width layer in the set of low-bit-width layers, calculating a per-layer deviation statistic, in a set of per-layer deviation statistics, of the low-bit-width layer based on a mean squared error of a difference between an example output activation of a floating-point layer, in the set of floating-point layers, and a low-bit-width output activation of the low-bit-width layer;

generating a quantized network representing the floating-point network and comprising the set of low-bit-width layers; and

in response to an accuracy of the quantized network falling below a loss-of-accuracy threshold:

converting a floating-point layer, represented by a first low-bit-width layer exhibiting a greatest per-layer deviation statistic in the set of per-layer deviation statistics, to a high-bit-width layer; and

replacing the first low-bit-width layer with the high-bit-width layer in the quantized network.

16. The method of claim 15 , further comprising:

for each floating-point layer in the set of floating-point layers, calculating the example output activation of the floating-point layer based on a validation example; and

for each low-bit-width layer in the set of low-bit-width layers, calculating the low-bit-width output activation of the low-bit-width layer based on an example input activation.

17. The method of claim 15 :

further comprising:

calculating a first accuracy of the floating-point network;

for each floating-point layer in the set of floating point layers:

calculating a second accuracy of the floating-point network based on a low-bit-width layer, corresponding to the floating-point layer, and the set of floating point layers excluding the floating point layer; and

calculating a layer priority for the low-bit-width layer corresponding to the floating-point layer based on a difference between the second accuracy and the first accuracy; and

for each low-bit-width layer in the set of low-bit-width layers, sorting the low-bit-width layer in the set of low-bit-width layers based on the per-layer deviation statistic of the low-bit-width layer and the layer priority for the low-bit-width layer as a set of ordered low-bit-width layers; and

wherein converting the floating-point layer comprises converting the floating-point layer, represented by the first low-bit-width layer in the set of ordered low-bit-width layers, to the high-bit-width layer.

18. The method of claim 15 :

further comprising, for each low-bit-width layer in the set of low-bit-width layers, sorting the low-bit-width layer in the set of low-bit-width layers, as a set of ordered low-bit-width layers, based on the per-layer deviation statistic and a layer type of the low-bit-width layer; and

wherein converting the floating-point layer comprises converting the floating-point layer, represented by the first low-bit-width layer in the set of ordered low-bit-width layers, to the high-bit-width layer.

19. A method for quantizing an artificial neural network, the method comprising:

converting a set of floating-point layers in a floating-point network to a set of low-bit-width layers;

for each low-bit-width layer in the set of low-bit-width layers, calculating a per-layer deviation statistic, in a set of per-layer deviation statistics, of the low-bit-width layer based on a set of error metrics between a set of low-bit-width output activations of the low-bit-width layer and a set of example output activations of a floating-point layer, each error metric in the set of error metrics characterized by a difference between an example output activation in the set of example output activations and a corresponding low-bit-width output activation in the set of low-bit-width output activations, wherein the per-layer deviation statistic is calculated based on a mean squared error of the error metric for each example output activation in the set of example output activations;

generating a quantized network representing the floating-point network and comprising the set of low-bit-width layers; and

in response to an accuracy of the quantized network falling below a loss-of-accuracy threshold:

converting a floating-point layer, represented by a low-bit-width layer exhibiting a greatest per-layer deviation statistic in the set of per-layer deviation statistics, to a high-bit-width layer; and

replacing the low-bit-width layer with the high-bit-width layer in the quantized network.

20. The method of claim 19 , wherein calculating the per-layer deviation statistic comprises:

for each example output activation in the set of example output activations, calculating an error metric between a corresponding eight-bit output activation, in the set of eight-bit output activations, and the example output activation.