System and method for adaptive graph-to-stream scheduling
This application describes an accelerator, a computer system, and a method for adaptive graph-to-stream scheduling in processors. An example method may include receiving a computation graph for a GPU, the computation graph comprising (1) a plurality of nodes representing a plurality of kernels for the GPU to execute and (2) a plurality of edges representing execution dependencies among the plurality of kernels; performing one or more wave partitions on the computation graph to determine a plurality of waves of kernels; obtaining a kernel resource table comprising resource usage of each kernel; mapping the plurality of kernels into a plurality of streams based on the plurality of waves and the kernel resource table; and executing the plurality of streams on the GPU, wherein kernels mapped in a same stream are executed sequentially by the GPU, and kernels mapped to different streams are concurrently executable by the GPU.
1 . A computer-implemented method, comprising:
receiving a computation graph for a graphics processing unit (GPU), the computation graph comprising (1) a plurality of nodes representing a plurality of kernels for the GPU to execute and (2) a plurality of edges representing execution dependencies among the plurality of kernels;
performing one or more wave partitions on the computation graph to determine a plurality of waves of kernels, each wave comprising one or more kernels that are concurrently executable on the GPU;
obtaining a kernel resource table comprising resource usage of each kernel;
mapping the plurality of kernels into a plurality of streams based on the plurality of waves and the kernel resource table; and
executing the plurality of streams on the GPU, wherein kernels mapped in a same stream are executed sequentially by the GPU, and kernels mapped to different streams are concurrently executable by the GPU.
2 . The method of claim 1 , wherein the mapping the plurality of kernels into the plurality of streams based on the plurality of waves and the kernel resource table comprises:
for the one or more kernels from the same wave of the plurality of waves, identifying one or more kernels with resource usage being greater than a threshold;
assigning the one or more identified kernels to the same stream; and
assigning other kernels from the same wave to the plurality of streams in a round-robin manner.
3 . The method of claim 1 , wherein the obtaining the kernel resource table comprising the resource usage of each kernel comprises:
obtaining the kernel resource table from a compiler compiling the plurality of kernels.
4 . The method of claim 1 , wherein the performing wave partitions on the computation graph to determine the plurality of waves of kernels comprises:
performing breadth-first search (BFS) on the computation graph to determine a depth for each of the plurality of nodes in the computation graph; and
determining one or more kernels from a same depth as a wave of kernels.
5 . The method of claim 4 , wherein the performing BFS on the computation graph to determine the depth for each of the plurality of nodes comprises:
when one node appears on more than one paths in the computation graph, determining, based on BFS, more than one depths of the node respectively corresponding to the more than one paths; and
determining a maximum depth of the more than one depths as a depth of the node.
6 . The method of claim 1 , wherein the plurality of streams comprise Compute Unified Device Architecture (CUDA) streams.
7 . The method of claim 1 , further comprising:
inserting one or more synchronization kernels into one or more of the streams to maintain kernel dependency correctness between kernels.
8 . The method of claim 7 , wherein the inserting the one or more synchronization kernels comprises:
when a kernel at a front of a first stream depends on an execution of a kernel at a front of a second stream, inserting a synchronization kernel before the kernel at the front of the first stream so that the GPU executes the kernel at the front of the first stream only when the kernel at the front of a second stream is executed.
9 . The method of claim 1 , further comprising:
detecting load imbalance among the plurality of streams periodically; and
remapping a kernel from one stream to another stream to balance load among the plurality of streams.
10 . The method of claim 9 , wherein:
the computation graph is executed by the GPU for a plurality of iterations, and
the remapping of the kernel from one stream to another stream affects a current iteration of executing the computation graph and not future iterations.
11 . The method of claim 1 , wherein the executing the plurality of streams on the GPU comprises:
fetching kernels from heads of the plurality of streams; and
dispatching the fetched kernels to a plurality of Streaming Multiprocessors (SMs) for parallel processing.
12 . The method of claim 11 , wherein the fetching the kernels from the heads of the plurality of streams comprises:
fetching kernels from the heads of the plurality of streams in a round-robin manner.
13 . A hardware accelerator for automatic and adaptive graph-to-stream scheduling for graphics processing units (GPUs), comprising:
a graph wave partition circuitry configured to:
receive a computation graph for a GPU, the computation graph comprising (1) a plurality of nodes representing a plurality of kernels for the GPU to execute and (2) a plurality of edges representing execution dependencies among the plurality of kernels; and
perform one or more wave partitions on the computation graph to determine a plurality of waves of kernels, each wave comprising one or more kernels that are concurrently executable on the GPU;
a kernel resource usage circuitry configured to:
obtain a kernel resource table comprising resource usage of each kernel;
a kernel-stream mapping circuitry configured to:
map the plurality of kernels into a plurality of streams based on the plurality of waves and the kernel resource table; and
a global dispatcher configured to:
dispatch the mapped kernels in the plurality of streams into a plurality of Streaming Multiprocessors (SMs) for parallel processing.
14 . The hardware accelerator of claim 13 , wherein the kernel resource usage circuitry is further configured to:
obtain the kernel resource table from a compiler compiling the plurality of kernels.
15 . The hardware accelerator of claim 13 , wherein the graph wave partition circuitry is further configured to:
perform breadth-first search (BFS) on the computation graph to determine a depth for each of the plurality of nodes in the computation graph; and
determine kernels from a same depth as a wave of kernels.
16 . The hardware accelerator of claim 13 , wherein the plurality of streams comprise Compute Unified Device Architecture (CUDA) streams.
17 . The hardware accelerator of claim 13 , further comprising:
a sync insertion circuitry configured to:
when a kernel at a front of a first stream depends on an execution of a kernel at a front of a second stream, insert a synchronization kernel before the kernel at the front of the first stream so that the GPU executes the kernel at the front of the first stream when the kernel at the front of a second stream is executed.
18 . The hardware accelerator of claim 13 , further comprising:
a kernel re-mapping circuitry configured to:
detect load imbalance among the plurality of streams periodically; and
remap a kernel from one stream to another stream to balance load among the plurality of streams.
19 . The hardware accelerator of claim 18 , wherein the computation graph is being executed by the GPU for a plurality of iterations, and
the remapping of the kernel from one stream to another stream only affects a current iteration of executing the computation graph but not future iterations.
20 . A non-transitory computer-readable storage medium, the storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving a computation graph for a graphics processing unit (GPU), the computation graph comprising (1) a plurality of nodes representing a plurality of kernels for the GPU to execute and (2) a plurality of edges representing execution dependencies among the plurality of kernels;
performing one or more wave partitions on the computation graph to determine a plurality of waves of kernels, each wave comprising one or more kernels that are concurrently executable on the GPU;
obtaining a kernel resource table comprising resource usage of each kernel;
mapping the plurality of kernels into a plurality of streams based on the plurality of waves and the kernel resource table; and
executing the plurality of streams on the GPU, wherein kernels mapped in a same stream are executed sequentially by the GPU, and kernels mapped to different streams are concurrently executable by the GPU.