SYSTEM AND METHOD FOR SCHEDULING IN A COMPUTING SYSTEM
An improved multi-level scheduling system and method are disclosed. In one embodiment, the system comprises a coarse scheduler to allocate sets of computing resources at a first level and a set of fine grain schedulers configured to schedule at a second level, wherein the second level comprises individual computing resources within each set of computing resources. The fine grain scheduler may be configured to communicate with the coarse scheduler and monitor performance and utilization of the individual computing resources. The fine grain schedulers may also be configured to implement a different set of allocation rules than the coarse scheduler and request additional sets of resources from the coarse scheduler based on current and predicted utilization of the individual computing resources.
1 . A method for scheduling tasks in a computing system, the method comprising:
prompting a user to specify an application to be run;
prompting the user to specify a data source to be processed by the application;
prompting the user to specify a location for the application to store results;
creating a coarse scheduler configured to allocate container level resources; and
creating one or more containers for the application, wherein each container comprises a fine grain scheduler configured to schedule in-container processes.
2 . The method of claim 1 , wherein the fine grain scheduler is configured to communicate with the coarse scheduler.
3 . The method of claim 1 , wherein the fine grain scheduler is configured to implement a different set of allocation rules than the coarse scheduler.
4 . The method of claim 1 , further comprising creating a set of one or more pods for the application, wherein each of the containers is contained within one of the pods.
5 . The method of claim 1 , wherein the coarse scheduler is configured to share computing system resources between pods.
6 . The method of claim 5 , wherein the computing system resources comprise CPUs and GPUs.
7 . The method of claim 1 , wherein the coarse scheduler is configured based on historical performance data collected from prior runs of the application.
8 . The method of claim 1 , wherein the fine grain scheduler is configured based on historical performance data collected from prior runs of the application.
9 . The method of claim 1 , further comprising creating a plurality of coarse host processes and a plurality of fine grain processes.
10 . A non-transitory, computer-readable storage medium storing instructions executable by a processor of a computational device, which when executed cause the computational device to:
operate a coarse scheduler configured to allocate coarse blocks of resources to portions of an application that is to be executed; and
operate a fine-grained scheduler configured to (i) allocate tasks to queues, (ii) assign nodes to the queues with tasks, and (iii) monitor queue length and resource utilization of the assigned nodes, wherein if the queue length or resource utilization are above a first predetermined threshold, the fine-grained scheduler is configured to request additional resources from the coarse scheduler.
11 . The non-transitory, computer-readable storage medium of claim 10 , wherein the fine-grained scheduler is further configured to request additional coarse blocks of resources if available resources fall below a second predetermined threshold.
12 . The non-transitory, computer-readable storage medium of claim 10 , wherein the fine-grained scheduler is further configured to applies a resource allocation policy based on resource availability and capabilities.
13 . The non-transitory, computer-readable storage medium of claim 10 , wherein the fine-grained scheduler applies a resource allocation policy based on resource availably and capabilities.
14 . The non-transitory, computer-readable storage medium of claim 10 , wherein the fine-grained scheduler applies a resource allocation policy further based on per description or historical/prediction data.
15 . A method for scheduling tasks in a computing system, the method comprising:
allocating coarse blocks of resources to portions of an application that is to be executed;
allocating tasks to queues;
assigning nodes to the queues with allocated tasks; and
monitoring queue length and resource utilization of the assigned nodes, wherein if the queue length or allocated resource utilization are above a first predetermined threshold, allocating additional resources from a coarse scheduler.
16 . The method of claim 15 , further comprising requesting additional coarse blocks of resources if available resources fall below a second predetermined threshold.
17 . The method of claim 16 , further comprising:
predicting when additional coarse blocks of resources may be needed; and
performing said requesting in response thereto.
18 . The method of claim 16 , further comprising monitoring utilization levels.
19 . The method of claim 18 , further comprising:
predicting when additional coarse blocks of resources may be needed based on current and historical utilization levels; and
performing said requesting in response thereto.