Static scheduling and dynamic scheduling for compiler-hinted and self-scheduling multi-engine artificial intelligence (AI) processing unit system
Aspects of the present disclosure provide an apparatus. For example, the apparatus can include a compiler configured to compile a neural network (NN) model to generate a plurality of operations/threads and determine whether each of the operations/threads is compute bound or memory bound, and a memory coupled to the compiler and configured to store the operations/threads. The apparatus can also include a thread scheduler coupled to the memory and configured to schedule the operations/threads of the NN model. The apparatus can also include a multi-engine processing unit that includes a plurality of compute units (CUs), and an executor coupled between the thread scheduler and the multi-engine processing unit. The executor can be configured to allocate the operations/threads of the NN model and activate a number of the CUs of the multi-engine processing unit for each of the operations/threads based on whether the operation/thread is compute bound or memory bound.
1 . An apparatus for optimizing resource allocation in a multi-engine artificial intelligence (AI) processing unit, comprising:
a compiler configured to compile a neural network (NN) model to generate a plurality of operations assigned to threads for execution, and determine whether each of the operations and threads is compute bound or memory bound;
a memory coupled to the compiler, the memory configured to store the operations and threads of the NN model;
a thread scheduler coupled to the memory, the thread scheduler configured to schedule the operations and threads of the NN model;
a multi-engine AI processing unit that includes a plurality of physical compute units (CUs); and
an executor coupled between the thread scheduler and the multi-engine AI processing unit, the executor configured to allocate the operations and threads of the NN model and activate a number of the physical CUs of the multi-engine AI processing unit for each of the operations and threads, based on runtime performance of the multi-engine AI processing unit, and based on whether the operation and/or thread is compute bound or memory bound.
2 . The apparatus of claim 1 , further comprising:
a performance monitor coupled to the executor, the performance monitor configured to monitor runtime performance of the multi-engine processing unit and/or runtime performance of a network coupled to the multi-engine processing unit,
wherein the executor is further configured to change the number of the CUs that are activated based on the runtime performance of the multi-engine processing unit and/or the runtime performance of the network.
3 . The apparatus of claim 2 , wherein the executor is configured to change the number of the physical CUs that are activated by activating one of the physical CUs that are not activated or deactivating one of the physical CUs that are activated at a time.
4 . The apparatus of claim 2 , wherein the runtime performance of the multi-engine processing unit includes throughput of the physical CUs.
5 . The apparatus of claim 2 , wherein the runtime performance of the network includes input/output bandwidth between the network and the multi-engine processing unit.
6 . The apparatus of claim 1 , wherein the multi-engine processing unit further includes a buffer that is coupled to and shared by at least two of the physical CUs.
7 . The apparatus of claim 1 , wherein the thread scheduler is configured to schedule the operation and threads by maximizing uRate of the multi-engine processing unit while meeting thread execution constraints.
8 . The apparatus of claim 1 , wherein the compiler determines that one of the operations and threads is the compute bound if a compute cycle of the operation or thread is greater than a memory read/write (R/W) cycle of the operation or thread, or is the memory bound if the compute cycle is not greater than the memory R/W cycle.
9 . The apparatus of claim 1 , wherein the memory includes a queue.
10 . An apparatus for optimizing resource allocation in a multi-engine artificial intelligence (AI) processing unit, comprising:
a compiler configured to compile a neural network (NN) model to generate a plurality of operations assigned to threads for execution;
a memory coupled to the compiler, the memory configured to store the operations and threads of the NN model;
a thread scheduler coupled to the memory, the thread scheduler configured to schedule the operations and threads of the NN model;
a multi-engine AI processing unit that includes a plurality of physical compute units (CUs);
a performance monitor configured to monitor runtime performance of the multi-engine AI processing unit and/or runtime performance of a network coupled to the multi-engine AI processing unit; and
an executor coupled between the thread scheduler and the multi-engine AI processing unit, and the performance monitor, the executor configured to allocate the operations and threads of the NN model, activate a number of the physical CUs of the multi-engine AI processing unit for each of the operations and threads, and change the number of physical CUs that are activated based on the runtime performance of the multi-engine AI processing unit and/or the runtime performance of the network, and based on whether the operation and/or thread is determined to be compute bound or memory bound.
11 . The apparatus of claim 10 , wherein the executor is configured to change the number of the physical CUs that are activated by activating one of the physical CUs that are not activated or deactivating one of the physical CUs that are activated at a time.
12 . The apparatus of claim 10 , wherein the runtime performance of the multi-engine processing unit includes throughput of the physical CUs.
13 . The apparatus of claim 10 , wherein the runtime performance of the network includes input/output bandwidth between the network and the multi-engine processing unit.
14 . The apparatus of claim 10 , wherein the compiler is further configured to determine whether each of the operations and threads is compute bound or memory bound, and the executor is configured to activate the number of the physical CUs of the multi-engine processing unit for each of the operations and threads based on whether the operation or thread is compute bound or memory bound.
15 . The apparatus of claim 10 , wherein the multi-engine processing unit further includes a buffer that is coupled to and shared by at least two of the physical CUs.
16 . The apparatus of claim 10 , wherein the thread scheduler is configured to schedule the operation/threads by maximizing uRate of the multi-engine processing unit while meeting thread execution constraints.
17 . The apparatus of claim 10 , wherein the compiler determines that one of the operations and threads is the compute bound if a compute cycle of the operation or thread is greater than a memory read/write (R/W) cycle of the operation or thread, or is the memory bound if the compute cycle is not greater than the memory R/W cycle.
18 . The apparatus of claim 10 , wherein the memory includes a queue.