Method for automatic hybrid quantization of deep artificial neural networks

1. A method for quantizing artificial neural networks including:

accessing a floating-point network comprising a set of floating-point layers;

accessing a set of validation examples for the floating-point network;

for each floating-point layer in the set of floating-point layers:

calculating a set of example input activations of the floating-point layer based on the set of validation examples and a preceding subset of floating-point layers in the floating-point network;

calculating a set of example output activations of the floating-point layer based on the set of example input activations of the floating point layer;

converting the floating-point layer to a low-bit-width layer in a set of low-bit-width layers;

calculating a set of low-bit-width output activations of the low-bit-width layer based on the set of example input activations; and

calculating a per-layer deviation statistic of the low-bit-width layer based on a set of error metrics between the set of low-bit-width output activations of the low-bit-width layer and the set of example output activations of the floating-point layer, each error metric in the set of error metrics characterized by a difference between an example output activation in the set of example output activations and a corresponding low-bit-width output activation in the set of low bit-width activations;

generating a quantized network representing the floating-point network and comprising the set of low-bit-width layers;

calculating an accuracy of the quantized network based on the set of validation examples; and

while a loss-of-accuracy threshold exceeds the accuracy of the quantized network, sequentially, from a greatest-deviating low-bit-width layer in the set of low-bit-width layers toward a least-deviating low-bit-width layer in the set of low-bit-width layers:

converting a floating-point layer represented by the low-bit-width layer to a high-bit-width layer;

replacing the low-bit-width layer with the high-bit-width layer in the quantized network;

updating the accuracy of the quantized network based on the set of validation examples; and

in response to the accuracy of the quantized network exceeding the loss-of-accuracy threshold, returning the quantized network.

2. The method of claim 1 :

wherein converting the floating-point layer to the low-bit-width layer in the set of low-bit-width layers comprises converting the floating-point layer to an eight-bit layer in a set of eight-bit layers;

wherein calculating the set of low-bit-width output activations of the low-bit-width layer based on the set of example input activations comprises calculating a set of eight-bit output activations of the eight-bit layer based on the set of example input activations;

wherein calculating the per-layer deviation statistic of the low-bit-width layer based on the set of error metrics between the set of low-bit-width output activations of the low-bit-width layer and the set of example output activations of the floating-point layer comprises calculating a per-layer deviation statistic of the eight-bit layer based on a set of error metrics between the set of eight-bit output activations of the eight-bit layer and the set of example output activations of the floating-point layer;

wherein generating the quantized network representing the floating-point network and comprising the set of low-bit-width layers comprises generating the quantized network representing the floating-point network and comprising the set of eight-bit layers; and

further comprising, while the loss-of-accuracy threshold exceeds the accuracy of the quantized network, sequentially, from a greatest-deviating eight-bit layer in the set of eight-bit layers toward a least-deviating eight-bit layer in the set of eight-bit layers:

converting a floating-point layer represented by the eight-bit layer to a sixteen-bit layer;

replacing the eight-bit layer with the sixteen-bit layer in the quantized network;

updating the accuracy of the quantized network based on the set of validation examples; and

in response to the accuracy of the quantized network exceeding the loss-of-accuracy threshold, returning the quantized network.

3. The method of claim 1 , further comprising, in response to the accuracy of the quantized network exceeding the loss-of-accuracy threshold, loading the quantized network onto an edge device.

4. A method for quantizing artificial neural networks including:

accessing a floating-point network comprising a set of floating-point layers;

accessing a set of validation examples for the floating-point network;

for each floating-point layer in the set of floating-point layers:

calculating a set of example input activations of the floating-point layer and a set of example output activations of the floating-point layer based on the set of validation examples;

converting the floating-point layer to a low-bit-width layer in a set of low-bit-width layers;

calculating a set of low-bit-width output activations of the low-bit-width layer based on the set of example input activations; and

calculating a per-layer deviation statistic of the low-bit-width layer based on a set of error metrics between the set of low-bit-width output activations of the low-bit-width layer and the set of example output activations of the floating-point layer, each error metric in the set of error metrics characterized by a difference between an example output activation in the set of example output activations and a corresponding low-bit-width output activation in the set of low bit-width activations;

generating a quantized network representing the floating-point network and comprising the set of low-bit-width layers;

calculating an accuracy of the quantized network based on the set of validation examples;

ordering the set of low-bit-width layers based on the per-layer deviation statistic of each low-bit-width layer in the set of low-bit-width layers to generate an ordered set of low-bit-width layers; and

while a loss-of-accuracy threshold exceeds the accuracy of the quantized network, sequentially, in the ordered set of low-bit-width layers:

converting a floating-point layer represented by the low-bit-width layer to a high-bit-width layer;

replacing the low-bit-width layer with the high-bit-width layer in the quantized network;

updating the accuracy of the quantized network based on the set of validation examples; and

in response to the accuracy of the quantized network exceeding the loss-of-accuracy threshold, returning the quantized network.

5. The method of claim 4 :

wherein converting the floating-point layer to the low-bit-width layer in the set of low-bit-width layers comprises converting the floating-point layer to an eight-bit layer in a set of eight-bit layers;

wherein calculating the set of low-bit-width output activations of the low-bit-width layer based on the set of example input activations comprises calculating a set of eight-bit output activations of the eight-bit layer based on the set of example input activations;

wherein calculating the per-layer deviation statistic of the low-bit-width layer based on the set of error metrics between the set of low-bit-width output activations of the low-bit-width layer and the set of example output activations of the floating-point layer comprises calculating a per-layer deviation statistic of the eight-bit layer based on a set of error metrics between the set of eight-bit output activations of the eight-bit layer and the set of example output activations of the floating-point layer;

wherein generating the quantized network representing the floating-point network and comprising the set of low-bit-width layers comprises generating the quantized network representing the floating-point network and comprising the set of eight-bit layers;

wherein ordering the set of low-bit-width layers based on the per-layer deviation statistic of each low-bit-width layer in the set of low-bit-width layers to generate the ordered set of low-bit-width layers comprises ordering the set of eight-bit layers based on the per-layer deviation statistic of each eight-bit layer in the set of eight-bit layers to generate an ordered set of eight-bit layers; and

further comprising, while the loss-of-accuracy threshold exceeds the accuracy of the quantized network, sequentially, in the ordered set of eight-bit layers:

converting a floating-point layer represented by the eight-bit layer to a sixteen-bit layer;

replacing the eight-bit layer with the sixteen-bit layer in the quantized network;

updating the accuracy of the quantized network based on the set of validation examples; and

in response to the accuracy of the quantized network exceeding the loss-of-accuracy threshold, returning the quantized network.

6. The method of claim 4 , wherein ordering the set of low-bit-width layers based on the per-layer deviation statistic of each low-bit-width layer in the set of low-bit-width layers to generate the ordered set of low-bit-width layers comprises ordering the set of low-bit-width layers, based on the per-layer deviation statistic of each low-bit-width layer in the set of low-bit-width layers and a network position of each low-bit-width layer in the set of low-bit-width layers, to generate the ordered set of low-bit-width layers.

7. The method of claim 6 , wherein ordering the set of low-bit-width layers, based on the per-layer deviation statistic of each low-bit-width layer in the set of low-bit-width layers and the network position of each low-bit-width layer in the set of low-bit-width layers, to generate the ordered set of low-bit-width layers comprises:

selecting a highest-deviating subset of the set of low-bit-width layers based on the per-layer deviation statistic of each low-bit-width layer in the set of low-bit-width layers; and

ordering the highest-deviating subset from an earliest network position to a latest network position based on the network position of each low-bit-width layer in the set of low-bit-width layers to generate the ordered set of low-bit-width layers.

8. The method of claim 4 :

further comprising:

calculating a first accuracy of the floating-point network based on the validation examples; and

for each floating-point layer in the set of floating point layers:

calculating a second accuracy of the floating-point network based on:

a low-bit-width layer corresponding to the floating-point layer and the set of floating point layers excluding the floating point layer; and

the set of validation examples; and

calculating a layer priority for the low-bit-width layer corresponding to the floating-point layer based on a difference between the second accuracy and the first accuracy; and

wherein ordering the set of low-bit-width layers based on the per-layer deviation statistic of each low-bit-width layer in the set of low-bit-width layers comprises ordering the set of low-bit-width layers, to generate the ordered set of low-bit-width layers, based on:

the per-layer deviation statistic of each low-bit-width layer in the set of low-bit-width layers; and

the layer priority of each low-bit-width layer in the set of low-bit-width layers.

9. The method of claim 8 , wherein ordering the set of low-bit-width layers, to generate the ordered set of low-bit-width layers, based on the per-layer deviation statistic of each low-bit-width layer in the set of low-bit-width layers and the layer priority of each low-bit-width layer in the set of low-bit-width layers comprises:

ordering the set of low-bit-width layers based on the layer priority of each low-bit-width layer in the set of low-bit-width layers to generate a second ordered set of low-bit-width layers;

selecting a subset of low-bit-width layers in the second ordered set of low-bit-width layers; and

ordering the subset of low-bit-width layers based on the per-layer deviation statistic of each low-bit-width layer in the subset of low-bit-width layers to generate the ordered set of low-bit-width layers.

10. The method of claim 4 , wherein converting the floating-point layer to the low-bit-width layer in the set of low-bit-width layers comprises, for each floating-point channel of the floating-point layer, converting the floating-point channel to a low-bit-width channel in the low-bit-width layer in the set of low-bit-width layers.

11. The method of claim 4 , wherein converting the floating-point layer to the low-bit-width layer in the set of low-bit-width layers comprises, for each floating-point output channel of the floating-point layer, converting the floating-point output channel to a low-bit-width output channel in the low-bit-width layer in the set of low-bit-width layers.

12. The method of claim 4 , wherein calculating the per-layer deviation statistic of the low-bit-width layer based on the set of error metrics between the set of low-bit-width output activations of the low-bit-width layer and the set of example output activations of the floating-point layer comprises:

for each example output activation in the set of example output activations, calculating an error metric between a corresponding low-bit-width output activation in the set of low-bit-width output activations and the example output activation; and

calculating the per-layer deviation statistic equal to a mean squared error of the error metric for each example output activation in the set of example output activations.

13. The method of claim 4 , wherein converting the floating-point layer to the low-bit-width layer in the set of low-bit-width layers comprises converting the floating-point layer to the low-bit-width layer in the set of low-bit-width layers, the low-bit-width layer represented in Q-format fixed-point notation.

14. The method of claim 13 , wherein converting the floating-point layer to the low-bit-width layer in the set of low-bit-width layers comprises:

converting a set of floating-point weights of the floating-point layer to a set of Q-format fixed-point weights of the low-bit-width layer in the set of low-bit-width layers;

converting a bias of the floating-point layer to a Q-format fixed-point bias of the low-bit-width layer in the set of low-bit-width layers; and

bit-shifting the set of Q-format fixed-point weights to match the Q-format fixed-point bias.

15. The method of claim 4 , further comprising, in response to the accuracy of the quantized network exceeding the loss-of-accuracy threshold, loading the quantized network onto an edge device.

16. The method of claim 4 , further comprising:

calculating a performance metric of the quantized network based on the set of validation examples; and

while the loss-of-accuracy threshold exceeds the accuracy of the quantized network, sequentially, in the ordered set of low-bit-width layers, updating the performance metric of the quantized network based on the set of validation examples.

17. The method of claim 16 , further comprising, while the loss-of-accuracy threshold exceeds the accuracy of the quantized network and while the performance metric of the quantized network exceeds a performance metric threshold, sequentially, in the ordered set of low-bit-width layers:

converting the floating-point layer represented by the low-bit-width layer to the high-bit-width layer;

replacing the low-bit-width layer with the high-bit-width layer in the quantized network;

updating the accuracy of the quantized network based on the set of validation examples;

updating the performance metric of the quantized network based on the set of validation examples; and

in response to the accuracy of the quantized network exceeding the loss-of-accuracy threshold, returning the quantized network.

18. The method of claim 16 , further comprising rendering a plot of the accuracy of the quantized network and the performance metric of the quantized network for each replacement of a low-bit-width layer with a high-bit-width layer in the quantized network.

19. The method of claim 4 , further comprising, while the loss-of-accuracy threshold exceeds the accuracy of the quantized network, sequentially, from a greatest-deviating low-bit-width layer in the set of low-bit-width layers toward a least-deviating low-bit-width layer in the set of low-bit-width layers:

converting the floating-point layer represented by the low-bit-width layer to the high-bit-width layer;

replacing the low-bit-width layer with the high-bit-width layer in the quantized network;

updating the accuracy of the quantized network based on the set of validation examples; and

in response to the accuracy of the quantized network exceeding the loss-of-accuracy threshold, returning the quantized network.

20. A method for quantizing artificial neural networks including:

accessing a floating-point network comprising a set of floating-point layers;

accessing a set of validation examples for the floating-point network;

for each floating-point layer in the set of floating-point layers:

calculating a set of example input activations of the floating-point layer and a set of example output activations of the floating-point layer based on the set of validation examples;

converting the floating-point layer to an eight-bit layer in a set of eight-bit layers;

calculating a set of eight-bit output activations of the eight-bit layer based on the set of example input activations;

for each example output activation in the set of example output activations, calculating an error metric, in a set of error metrics, characterized by a difference between the example output activation and a corresponding eight-bit output activation in the set of eight-bit output activations; and

calculating a per-layer deviation statistic of the eight-bit layer, based on the set of error metrics, characterizing a deviation between the set of eight-bit output activations of the eight-bit layer and the set of example output activations of the floating-point layer;

generating a quantized network representing the floating-point network and comprising the set of eight-bit layers;

calculating an accuracy of the quantized network based on the set of validation examples;

ordering the set of eight-bit layers based on the per-layer deviation statistic of each eight-bit layer in the set of eight-bit layers to generate an ordered set of eight-bit layers; and

while a loss-of-accuracy threshold exceeds the accuracy of the quantized network, sequentially, in the ordered set of eight-bit layers:

converting a floating-point layer represented by the eight-bit layer to a sixteen-bit layer;

replacing the eight-bit layer with the sixteen-bit layer in the quantized network;

updating the accuracy of the quantized network based on the set of validation examples; and

in response to the accuracy of the quantized network exceeding the loss-of-accuracy threshold, loading the quantized network onto an edge device.