Multilayered resource scheduling
Systems and methods for multilayered resource scheduling are provided. A system may include a user interface, an upper level scheduler, at least one lower level scheduler, and a reporting module. The user interface may receive a plurality of tasks. The upper level scheduler may sort the plurality of tasks into tasks with identical resource requirements. The lower level scheduler may be operable to receive resource requirements associated with a task. The lower level scheduler may ascertain compute resources sufficient to execute the task on a machine in a compute farm and advertise the compute resources as matching the resource requirements. The lower level scheduler may further receive a request to execute the task on the machine. The compute resources may be controlled to execute the tasks with the identical resource requirements without further scheduling operations. The lower level scheduler may control the compute resources to execute the task.
1. A method comprising:
receiving, by a first task scheduler, a plurality of tasks to be executed on a distributed computing system, each task having a respective set of resource requirements;
generating, using the first task scheduler, a proxy task comprising a subset of the tasks, the subset of the tasks having a common set of resource requirements;
scheduling, using a second task scheduler, an execution of the proxy task on the distributed computing system;
receiving, by the first task scheduler, an additional task to be executed on the distributed computing system, the additional task having the common set of resource requirements;
scheduling an execution of the additional task on the distributed computing system, wherein scheduling the execution of the additional task on the distributed computing system comprises including, by the first task scheduler, the additional task in the proxy task; and
performing at least one of:
discontinuing, by the first task scheduler, an inclusion of further tasks in the proxy task after a predetermined period of time after the generation of the proxy task, or
preventing an inclusion of further tasks in the proxy task by the first task schedule after a predetermined period of time after the generation of the proxy task.
2. The method of claim 1 , wherein scheduling the execution of the proxy task on the distributed computing system comprises:
identifying, by the second task scheduler, a set of resources of the distributed computing system satisfying the common set of resource requirements; and
controlling, by the second task scheduler, the set of resources to execute the proxy task.
3. The method of claim 1 , wherein the subset of the tasks are not visible to the second task scheduler as individual tasks.
4. A system comprising:
one or more processors; and
one or more non-transitory computer-readable media including one or more sequences of instructions which, when executed by the one or more processors, causes:
receiving, by a first task scheduler, a plurality of tasks to be executed on a distributed computing system, each task having a respective set of resource requirements;
generating, using the first task scheduler, a proxy task comprising a subset of the tasks, the subset of the tasks comprising at least two tasks of the plurality of tasks each having a common set of resource requirements;
scheduling, using a second task scheduler, an execution of the proxy task on the distributed computing system;
receiving, by the first task scheduler, an additional task to be executed on the distributed computing system, the additional task having the common set of resource requirements;
scheduling an execution of the additional task on the distributed computing system, wherein scheduling the execution of the additional task on the distributed computing system comprises including, by the first task scheduler, the additional task in the proxy task; and
performing at least one of:
discontinuing, by the first task scheduler, an inclusion of further tasks in the proxy task after a predetermined period of time after the generation of the proxy task, or
preventing an inclusion of further tasks in the proxy task by the first task schedule after a predetermined period of time after the generation of the proxy task.
5. The system of claim 4 , wherein scheduling the execution of the proxy task on the distributed computing system comprises:
identifying, by the second task scheduler, a set of resources of the distributed computing system satisfying the common set of resource requirements; and
controlling, by the second task scheduler, the set of resources to execute the proxy task.
6. The system of claim 4 , wherein the subset of the tasks are not visible to the second task scheduler as individual tasks.
7. One or more non-transitory computer-readable media including one or more sequences of instructions which, when executed by one or more processors, causes:
receiving, by a first task scheduler, a plurality of tasks to be executed on a distributed computing system, each task having a respective set of resource requirements;
generating, using the first task scheduler, a proxy task comprising a subset of the tasks, the subset of the tasks comprising at least two tasks of the plurality of tasks each having a common set of resource requirements;
scheduling, using a second task scheduler, an execution of the proxy task on the distributed computing system;
receiving, by the first task scheduler, an additional task to be executed on the distributed computing system, the additional task having the common set of resource requirements;
scheduling an execution of the additional task on the distributed computing system, wherein scheduling the execution of the additional task on the distributed computing system comprises including, by the first task scheduler, the additional task in the proxy task; and
performing at least one of:
discontinuing, by the first task scheduler, an inclusion of further tasks in the proxy task after a predetermined period of time after the generation of the proxy task, or
preventing an inclusion of further tasks in the proxy task by the first task schedule after a predetermined period of time after the generation of the proxy task.
8. The one or more non-transitory computer-readable media of claim 7 , wherein scheduling the execution of the proxy task on the distributed computing system comprises:
identifying, by the second task scheduler, a set of resources of the distributed computing system satisfying the common set of resource requirements; and
controlling, by the second task scheduler, the set of resources to execute the proxy task.
9. The one or more non-transitory computer-readable media of claim 7 , wherein the subset of the tasks are not visible to the second task scheduler as individual tasks.
10. The method of claim 1 , wherein the common set of resource requirements comprises at least one of:
a requirement for a particular number of central processing units (CPUs), or
a common requirement for a particular amount of memory.
11. A method comprising:
receiving, by a first task scheduler, a plurality of tasks to be executed on a distributed computing system, each task having a respective set of resource requirements;
generating, using the first task scheduler, a proxy task comprising a subset of the tasks, the subset of the tasks having a common set of resource requirements;
scheduling, using second task scheduler, an execution of the proxy task on the distributed computing system, wherein scheduling the execution of the proxy task on the distributed computing system comprises:
identifying, by the second task scheduler, a set of resources of the distributed computing system satisfying the common set of resource requirements, and
controlling, by the second task scheduler, the set of resources to execute the proxy task;
receiving, by the first task scheduler, an additional task to be executed on the distributed computing system, the additional task having the common set of resource requirements;
scheduling an execution of the additional task on the distributed computing system, wherein scheduling the execution of the additional task on the distributed computing system comprises including, by the first task scheduler, the additional task in the proxy task; and
discontinuing, by the first task scheduler, an inclusion of further tasks in the proxy task after a predetermined period of time after the generation of the proxy task.