Neural network computing system and method of executing neural network model
A neural network computing system includes a processor including heterogeneous computing devices configured to execute a neural network model; a memory configured to buffer input data and output data of the neural network model; a memory controller configured to control data input and data output of the memory; and a system bus configured to support communication between the processor and the memory controller. The processor determines a target execution time for each node included in the neural network model based on a target end-to-end execution time of the neural network model; controls operating frequencies of hardware devices including the heterogeneous computing devices, the memory controller, and the system bus based on a target computing device for execution of each node, an amount of work for each node, and the target execution time for each node, and executes the neural network model by operating at the operating frequencies.
1 . A neural network computing system, comprising:
a processor comprising heterogeneous computing devices configured to execute a neural network model;
a memory configured to buffer input data and output data of the neural network model;
a memory controller configured to control data input and data output of the memory; and
a system bus configured to support communication between the processor and the memory controller,
wherein the processor is configured to:
determine a target execution time for each node among a plurality of nodes comprised in the neural network model based on a target end-to-end execution time of the neural network model;
control respective operating clock frequencies of hardware devices, including the heterogeneous computing devices, the memory controller, and the system bus, based on a target computing device for execution of each node, an amount of work for each node, and the target execution time for each node; and
execute the neural network model by the hardware devices operating at the controlled respective operating clock frequencies, and
wherein the respective operating clock frequencies of the hardware devices are lowest operating clock frequencies of the hardware devices, respectively, at which the hardware devices perform a predetermined amount of work according to the target end-to-end execution time.
2 . The neural network computing system of claim 1 , wherein the processor is further configured to:
adjust the amount of work for each node and the target execution time for each node based on an actual end-to-end execution time of the neural network model; and
adjust an operating clock frequency of each of the hardware devices based on the adjusted amount of work for each node and the adjusted target execution time for each node.
3 . The neural network computing system of claim 1 , wherein the processor is further configured to:
determine an execution order of the plurality of nodes and whether to execute the plurality of nodes in parallel based on a connection relationship between the plurality of nodes and a plurality of edges included in the neural network model; and
determine the target execution time for each node by distributing the target end-to-end execution time of the neural network model to nodes, which are executed in sequence with respect to each other, in proportion to the amount of work for each node.
4 . The neural network computing system of claim 1 , wherein the processor is further configured to determine the amount of work for each node based on a number of arithmetic instructions of each node, a number of memory instructions of each node, a compiler option for each node, an amount of required memory bandwidth of each node, a size of input data for each node, and a size of output data for each node.
5 . The neural network computing system of claim 4 , wherein the processor is further configured to determine the number of arithmetic instructions and the number of memory instructions of each node by analyzing source code included in each node.
6 . The neural network computing system of claim 4 , wherein the processor is further configured to determine the amount of required memory bandwidth of each node based on the number of memory instructions, the size of the input data for each node, and the size of the output data for each node.
7 . The neural network computing system of claim 4 , wherein the amount of work for each node comprises the number of arithmetic instructions of each node, the number of memory instructions of each node, an amount of work of the target computing device for each node determined based on the compiler option, an amount of memory access for each node determined based on the amount of required memory bandwidth of each node, the size of the input data for each node, and the size of the output data for each node.
8 . The neural network computing system of claim 7 , wherein the processor is further configured to:
determine an amount of work of the heterogeneous computing devices required to execute the neural network model based on the amount of work of the target computing device for each node; and
determine an amount of work of the memory controller and an amount of work of the system bus required to execute the neural network model based on the amount of memory access for each node.
9 . The neural network computing system of claim 8 , wherein the processor is further configured to:
determine a minimum operating clock frequency at which the target execution time for each node is complied with from among a plurality of operating clock frequencies for each of the hardware devices based on a performance of each of the hardware devices and an amount of work for each of the hardware devices; and
control the respective operating clock frequencies of the hardware devices based on the determined minimum operating clock frequency of each of the hardware devices.
10 . The neural network computing system of claim 1 , wherein the heterogeneous computing devices comprise a central processing unit (CPU), a neural processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), and an accelerator.
11 . The neural network computing system of claim 1 , wherein the processor is further configured to receive the target end-to-end execution time of the neural network model as an input through an application programming interface (API).
12 . A neural network computing system comprising:
a processor comprising heterogeneous computing devices that are configured to execute a neural network model;
a memory configured to buffer input data and output data of the neural network model;
a memory controller configured to control data input and data output of the memory; and
a system bus configured to support communication between the processor and the memory controller,
wherein the processor comprises:
a feedforward control portion configured to:
estimate, based on an input of static analysis data that is independent of a result of execution of the neural network model, an amount of work of each node from among a plurality of nodes included in the neural network model; and
determine respective operating clock frequencies of hardware devices, including the heterogeneous computing devices, the system bus, and the memory controller, based on an input of a target end-to-end execution time of the neural network model and an input of the estimated amount of work of each node from among the plurality of nodes;
a neural network model execution portion configured to control the hardware devices according to the determined respective operating clock frequencies, execute the neural network model, and output an actual end-to-end execution time of the neural network model; and
a feedback control portion configured to adjust the respective operating clock frequencies of the hardware devices based on dynamic analysis data dependent on the result of the execution of the neural network model, including the actual end-to-end execution time, and
wherein the respective operating clock frequencies of the hardware devices are lowest operating clock frequencies of the hardware devices, respectively, at which the hardware devices perform a predetermined amount of work according to the target end-to-end execution time.
13 . The neural network computing system of claim 12 , wherein the feedback control portion is further configured to:
determine a degree of resource contention with an application simultaneously executed with the neural network model by analyzing a usage rate of each of the hardware devices while the neural network model is executed; and
adjust the respective operating clock frequencies of the hardware devices based on the degree of resource contention.
14 . The neural network computing system of claim 13 , wherein the feedback control portion is further configured to adjust the respective operating clock frequencies of the hardware devices to increase based on a usage rate occupied by the neural network model among usage rates of the hardware devices decreases.
15 . The neural network computing system of claim 13 ,
wherein the neural network computing system further comprises a camera,
wherein the application is a camera application generating an image frame using the camera, and
wherein the neural network model comprises at least one of a model for sensing an object in the image frame, and a model for identifying what the object is.
16 . The neural network computing system of claim 12 , wherein the feedback control portion is further configured to:
accumulate analysis data generated when the neural network model is repeatedly executed; and
generate the dynamic analysis data by processing the accumulated analysis data using one from among an average operation, an accumulative average operation, a simple moving average operation, a weighted moving average operation, and an exponential moving average operation of the accumulated analysis data.
17 . The neural network computing system of claim 12 , wherein the feedforward control portion is further configured to:
estimate an amount of work of a target computing device for execution of each node, among the plurality of nodes, and an amount of memory access for each node using the static analysis data;
determine a target execution time for each node based on the target end-to-end execution time of the neural network model;
estimate an amount of work of the heterogeneous computing devices based on the amount of work of the target computing device for each node;
estimate an amount of work of the memory controller and an amount of work of the system bus based on the amount of memory access for each node; and
determine an operating clock frequency of each of the hardware devices based on a performance of each of the hardware devices, the amount of work of the heterogeneous computing devices, the amount of work of the memory controller, and the amount of work of the system bus.
18 . The neural network computing system of claim 17 , wherein the feedforward control portion is further configured to determine an operating clock frequency, as the operating clock frequency of each of the hardware devices, at which the target execution time for each node is complied with and power consumption is minimized from among operating clock frequencies of the hardware devices.
19 . The neural network computing system of claim 12 , wherein the neural network model execution portion is further configured to release the operating clock frequencies of the hardware devices when execution of the neural network model is completed.
20 . A method of executing a neural network model, the method comprising:
estimating an amount of work for each node among a plurality of nodes included in the neural network model based on execution of the neural network model being triggered;
determining a target execution time for each node of the plurality of nodes based on a target end-to-end execution time of the neural network model;
controlling an operating clock frequency of each of a plurality of heterogeneous computing devices for executing the neural network model, based on a target computing device for each node, the amount of work for each node, and the target execution time for each node;
executing the neural network model using the plurality of heterogeneous computing devices operating according to the controlled operating clock frequency of each of the plurality of heterogeneous computing devices; and
adjusting the operating clock frequency of each of the plurality of heterogeneous computing devices for executing the neural network model based on an actual end-to-end execution time of the neural network model,
wherein the operating clock frequency is a lowest operating clock frequency of a respective one of the plurality of heterogeneous computing devices at which the respective one of the plurality of heterogeneous computing devices performs a predetermined amount of work according to the target end-to-end execution time.