Scheduling and execution of tasks based on resource availability
An apparatus for scheduling includes a receiving unit, configured to receive a task of a job; a configuration unit, configured to configure resource amounts for the task in at least two resource dimensions respectively, wherein each of the resource dimensions represents a resource available for configuring; and an interface unit, configured to distribute the task along with information about the resource amounts configured for the task in the resource dimensions.
1. An apparatus for scheduling, comprising:
a receiving unit configured to receive a task of a job;
a configuration unit configured to configure resource amounts for the task in at least two resource dimensions respectively, wherein each of the resource dimensions represents a resource available for configuring;
an interface unit configured to distribute the task along with information about the resource amounts configured for the task in the resource dimensions to available resources;
an executing unit configured to execute the task using the available resources;
a candidate unit configured to select tasks respectively from a plurality of jobs as candidate tasks to be distributed to the available resources, the receiving unit further configured to receive the candidate tasks, and the configuration unit further configured to configure resource amounts for each of the candidate tasks in at least two resource dimensions respectively, wherein each of the resource dimensions represents a resource available for configuring;
a decision unit configured to estimate both a gain and a cost of waiting to generate a scheme for distributing one or more of the candidate tasks to the available resources until after one or more currently executing tasks have completed;
a generation unit configured to generate a scheme for distributing the one or more of the candidate tasks to the available resources after the one or more currently executing tasks have completed according to the resource amounts configured for each of the one or more the candidate tasks in the resource dimensions when the estimated gain exceeds the estimated cost, the generation unit further configured to generate a scheme for distributing the one or more of the candidate tasks to the available resources while the one or more currently executing tasks are executing according to the resource amounts configured for each of the one or more candidate tasks in the resource dimensions when the estimated gain is less than or equal to the estimated cost, the interface unit further configured to distribute the one or more of the candidate tasks along with information about the resource amounts configured for the one or more of the candidate tasks to the available resources according to the generated scheme, and the executing unit further configured to execute the one or more of the candidate tasks using the available resources.
2. The apparatus according to claim 1 , further comprising: an estimation unit, configured to estimate the resources amounts resource amounts to be consumed by the task in the at least two resource dimensions, wherein the configuration unit configures the resource amounts for the task in at least two resource dimensions respectively according to result of the estimating.
3. The apparatus according to claim 2 , wherein the estimation unit estimates the resources amounts resource amounts to be consumed by the task in the at least two resource dimensions according to historical information.
4. The apparatus according to claim 2 , wherein the estimation unit estimates the resources amounts resource amounts to be consumed by the task in the at least two resource dimensions according to sampled information.
5. The apparatus according to claim 1 , wherein the apparatus of scheduling is included in a Map device of a MapReduce device.
6. The apparatus according to claim 1 , wherein the apparatus of scheduling is included in a Reduce device of a MapReduce device.