Systems and methods for quantization aware training of a neural network for heterogeneous hardware platform
Systems and methods are provided for quantization aware training of a neural network for heterogeneous hardware platform. In the method, the system acquires hardware profiles with respect to a plurality of hardware components of a heterogeneous hardware platform. The system determines a plurality of hardware configurations based on the hardware profiles. The system acquires a set of training data and performing a quantization aware training using the training data on a network model based on the hardware configurations. The system obtains the network model with model weights for the heterogeneous hardware platform.
1 . A method of performing quantization aware training (QAT) of a neural network, comprising:
acquiring hardware profiles with respect to a plurality of hardware components of a heterogeneous hardware platform, wherein the hardware profiles comprise performance data associated with the execution of different types of neural networks;
determining hardware configurations based on the performance data without imposing architecture constraints by:
selecting a computational component from the hardware components for each layer of the quantization aware training based on the hardware profiles comprising the performance data;
determining a computing precision for each layer of the quantization aware training based on the hardware profiles comprising the performance data; and
generating the hardware configurations associated with the selected computational component and the determined computing precision with respect to performing the neural network based on the hardware profile corresponding to the selected computational component, wherein the hardware configurations comprise one or more of quantization configurations and a plurality of computing precisions with respect to performing the neural network;
acquiring a set of training data and performing a quantization aware training using the training data on a network model based on the hardware configurations to mimic the hardware configurations of the heterogeneous hardware platform; and
obtaining the network model with model weights for the heterogeneous hardware platform based at least in part on the hardware configurations.
2 . The method of claim 1 ,
further comprising:
determining whether computational components and computing precisions are determined for each layer of quantization aware training;
in response to determining that the computational component and the computing precision are not determined for one layer of quantization aware training, selecting a computational component and determining a computing precision for a next layer of the quantization aware training based on the hardware profiles comprising the performance data; and
in response to determining that the computational component and the computing precision are determined for one layer of quantization aware training, acquiring the set of training data and performing the quantization aware training using the training data on the network model based on the hardware configurations.
3 . The method of claim 1 , wherein the plurality of computing precisions are determined based on the hardware profiles with respect to performing the neural network.
4 . The method of claim 1 , wherein the hardware configurations are determined based on:
selecting a computational component and determining a computing precision for each layer of the quantization aware training based on the hardware profiles, wherein the computational component is selected from the hardware components.
5 . The method of claim 1 , further comprising:
evaluating the trained network model on the heterogeneous hardware platform and obtaining an evaluation result; and
fine-tunning the hardware configurations based on the evaluation result.
6 . The method of claim 1 , wherein the hardware components are one or more selected from central processing unit (CPU), graphics processing unit (GPU), application-specific integrated circuit (ASIC), and field programmable gate array (FPGA).
7 . The method of claim 1 , wherein the performance data comprise throughput, latency, power consumption, or cost associated with the hardware components of the heterogeneous hardware platform.
8 . A quantization aware training (QAT) system, comprising:
at least one computer storage memory operable to store data along with computer-executable instructions; and
at least one processor operable to read the data and operate the computer-executable instructions to:
acquiring hardware profiles with respect to a plurality of hardware components of a heterogeneous hardware platform, wherein the hardware profiles comprise performance data associated with the execution of different types of neural networks;
determining hardware configurations based at least in part on the performance data without imposing architecture constraints by:
selecting a computational component from the hardware components for each layer of the quantization aware training based on the hardware profiles comprising the performance data;
determining a computing precision for each layer of the quantization aware training based on the hardware profiles comprising the performance data; and
generating the hardware configurations associated with the selected computational component and the determined computing precision with respect to performing the neural network based on the hardware profile corresponding to the selected computational component, wherein the hardware configurations comprise one or more of quantization configurations and a plurality of computing precisions with respect to performing the neural network;
acquiring a set of training data and performing a quantization aware training using the training data on a network model based on the hardware configurations to mimic hardware configurations of the heterogeneous hardware platform; and
outputting the network model with model weights for the heterogeneous hardware platform based at least in part on the hardware configurations.
9 . The QAT system of claim 8 , wherein the at least one processor is further configured to
determine whether computational components and computing precisions are determined for each layer of quantization aware training;
in response to determining that the computational component and the computing precision are not determined for one layer of quantization aware training, select a computational component and determine a computing precision for a next layer of the quantization aware training based on the hardware profiles comprising the performance data; and
in response to determining that the computational component and the computing precision are determined for one layer of quantization aware training, acquire the set of training data and perform the quantization aware training using the training data on the network model based on the hardware configurations.
10 . The QAT system of claim 8 , wherein the plurality of computing precisions are determined based on the hardware profiles with respect to performing the neural network.
11 . The QAT system of claim 8 , wherein the at least one processor is further configured to determine the hardware configurations based on:
selecting a computational component and determining a computing precision for each layer of the QAT based on the hardware profiles, wherein the computational component is selected from the hardware components.
12 . The QAT system of claim 8 , wherein the at least one processor is further configured to:
evaluating the trained network model on the heterogeneous hardware platform and obtaining an evaluation result; and
fine-tunning the hardware configurations based on the evaluation result.
13 . The QAT system of claim 8 , wherein the hardware components are one or more selected from central processing unit (CPU), graphics processing unit (GPU), application-specific integrated circuit (ASIC), and field programmable gate array (FPGA).
14 . The QAT system of claim 8 , wherein the performance data comprise throughput, latency, power consumption, or cost associated with the hardware components of the heterogeneous hardware platform.
15 . A non-transitory computer readable medium having stored thereon a program for executing a method of performing quantization aware training of a neural network, the method comprising:
acquiring hardware profiles with respect to a plurality of hardware components of a heterogeneous hardware platform, wherein the hardware profiles comprise performance data associated with the execution of different types of neural networks;
determining hardware configurations based at least in part on the performance data without imposing architecture constraints by:
selecting a computational component from the hardware components for each layer of the quantization aware training based on the hardware profiles comprising the performance data;
determining a computing precision for each layer of the quantization aware training based on the hardware profiles comprising the performance data; and
generating the hardware configurations associated with the selected computational component and the determined computing precision with respect to performing the neural network based on the hardware profile corresponding to the selected computational component, wherein the hardware configurations comprise one or more of quantization configurations and a plurality of computing precisions with respect to performing the neural network;
acquiring a set of training data and performing a quantization aware training using the training data on a network model based on the hardware configurations to mimic hardware configurations of the heterogeneous hardware platform; and
obtaining the network model with model weights for the heterogeneous hardware platform based at least in part on the hardware configurations.
16 . The non-transitory computer readable medium of claim 15 , wherein the method further comprises
determining whether computational components and computing precisions are determined for each layer of quantization aware training;
in response to determining that the computational component and the computing precision are not determined for one layer of quantization aware training, selecting a computational component and determining a computing precision for a next layer of the quantization aware training based on the hardware profiles comprising the performance data; and
in response to determining that the computational component and the computing precision are determined for one layer of quantization aware training, acquiring the set of training data and performing the quantization aware training using the training data on the network model based on the hardware configurations.
17 . The non-transitory computer readable medium of claim 15 , wherein the plurality of computing precisions are determined based on the hardware profiles with respect to performing the neural network.
18 . The non-transitory computer readable medium of claim 15 , wherein the method further determines the hardware configurations based on:
selecting a computational component and determining a computing precision for each layer of the QAT based on the hardware profiles, wherein the computational component is selected from the hardware components.
19 . The non-transitory computer readable medium of claim 15 , wherein the method further comprising:
evaluating the trained network model on the heterogeneous hardware platform and obtaining an evaluation result; and
fine-tunning the hardware configurations based on the evaluation result.
20 . The non-transitory computer readable medium of claim 15 , wherein the hardware profiles comprise throughput, latency, power consumption, or cost associated with the hardware components of the heterogeneous hardware platform.