Quantization-aware neural architecture search
Quantization-aware neural architecture search (“QNAS”) can be utilized to learn optimal hyperparameters for configuring an artificial neural network (“ANN”) that quantizes activation values and/or weights. The hyperparameters can include model topology parameters, quantization parameters, and hardware architecture parameters. Model topology parameters specify the structure and connectivity of an ANN. Quantization parameters can define a quantization configuration for an ANN such as, for example, a bit width for a mantissa for storing activation values or weights generated by the layers of an ANN. The activation values and weights can be represented using a quantized-precision floating-point format, such as a block floating-point format (“BFP”) having a mantissa that has fewer bits than a mantissa in a normal-precision floating-point representation and a shared exponent.
1. A computer-implemented method, comprising:
generating, by way of a recurrent neural network (RNN), hyperparameters for a child neural network, the hyperparameters comprising model topology parameters defining a structure and connectivity for an artificial neural network (ANN), quantization parameters defining a quantization configuration for the ANN, and hardware architecture parameters defining a number of tile engines to be utilized by a field-programmable gate array (FPGA) when executing the ANN;
configuring the child neural network according to the hyperparameters;
training the child neural network on a training data set;
following training of the child neural network, computing one or more metrics for the child neural network; and
performing reinforcement learning to train the RNN to generate second hyperparameters for a second child network using the one or more metrics as a reward signal for the RNN, the second hyperparameters comprising second model topology parameters, second quantization parameters, and second hardware architecture parameters.
2. The computer-implemented method of claim 1 , wherein the model topology parameters define a number of filters for the ANN.
3. The computer-implemented method of claim 1 , wherein the model topology parameters define a number of layers for the ANN.
4. The computer-implemented method of claim 1 , wherein the quantization parameters define a bit width for a mantissa for storing activation values generated by layers of the ANN.
5. The computer-implemented method of claim 4 , wherein the activation values are represented in a block floating-point format (BFP) comprising a mantissa and a shared exponent.
6. The computer-implemented method of claim 1 , wherein the quantization parameters define a bit width for a mantissa for storing weights for the ANN.
7. The computer-implemented method of claim 6 , wherein the weights are represented in a block floating-point format (BFP) having a mantissa and a shared exponent.
8. The computer-implemented method of claim 1 , wherein the one or more metrics comprise one or more of accuracy, inference time, or cost.
9. A computer-implemented method, comprising:
generating, by way of a recurrent neural network (RNN), hyperparameters for a child neural network, the hyperparameters comprising model topology parameters defining a structure and connectivity for an artificial neural network (ANN), quantization parameters defining a quantization configuration for the ANN, and hardware architecture parameters defining a number of tile engines to be utilized by a field-programmable gate array (FPGA) when executing the ANN;
configuring the child neural network according to the hyperparameters;
training the child neural network on a training data set;
following training of the child neural network, computing one or more metrics for the child neural network; and
performing reinforcement learning to train the RNN to generate second hyperparameters for a second child network using the one or more metrics as a reward signal for the RNN, the second hyperparameters comprising second model topology parameters, second quantization parameters, and second hardware architecture parameters.
10. The computer-implemented method of claim 9 , wherein the model topology parameters define a number of layers or a number of filters for the artificial neural network (ANN).
11. The computer-implemented method of claim 9 , wherein the quantization parameters define a bit width for a mantissa for storing activation values generated by layers of the ANN or a bit width for a mantissa for storing weights for the ANN.
12. The computer-implemented method of claim 11 , wherein the activation values and the weights are represented in a block floating-point format (BFP) having a mantissa and a shared exponent.
13. The computer-implemented method of claim 9 , wherein the one or more metrics comprise one or more of accuracy, inference time, or cost.
14. A computing device, comprising:
one or more processors; and
at least one computer storage media having computer-executable instructions stored thereupon which, when executed by the one or more processors, will cause the computing device to:
generate, by way of a recurrent neural network (RNN), hyperparameters for a child neural network, the hyperparameters comprising model topology parameters defining a structure and connectivity for an artificial neural network (ANN), quantization parameters defining a quantization configuration for the ANN, and hardware architecture parameters defining a number of tile engines to be utilized by a field-programmable gate array (FPGA) configured to execute the ANN;
configure the child neural network according to the hyperparameters;
train the child neural network on a training data set;
following training of the child neural network, compute one or more metrics for the child neural network; and
perform reinforcement learning to train the RNN to generate second hyperparameters for a second child network using the one or more metrics as a reward signal for the RNN, the second hyperparameters comprising second model topology parameters, second quantization parameters, and second hardware architecture parameters.
15. The computing device of claim 14 , wherein the quantization parameters define a bit width for a mantissa for storing activation values generated by layers of the ANN or a bit width for a mantissa for storing weights for the ANN.
16. The computing device of claim 15 , wherein the activation values and the weights are represented in a block floating-point format (BFP) having a mantissa and a shared exponent.
17. The computing device of claim 14 , wherein the model topology parameters define a number of layers or a number of filters for the artificial neural network (ANN).