Varying a characteristic of a job profile relating to map and reduce tasks according to a data size

1. A method of a system having a processor, comprising:

receiving a job profile that includes characteristics of a job to be executed, wherein the characteristics of the job profile relate to map tasks and reduce tasks of the job, wherein the map tasks produce intermediate results based on input data, and the reduce tasks produce an output based on the intermediate results, wherein the characteristics of the job profile include at least one particular characteristic that varies according to a size of data to be processed;

setting at least the particular characteristic of the job profile based on the size of the data to be processed;

providing a performance model based on the job profile containing the set particular characteristic and an allocated amount of resources for the job; and

estimating a performance parameter of the job using the performance model.

2. The method of claim 1 , wherein estimating the performance parameter comprises estimating a time duration of the job.

3. The method of claim 1 , further comprising:

computing at least one scaling factor according to the size of the data to be processed; and

applying the at least one scaling factor to the size of the data to be processed to set at least the particular characteristic.

4. The method of claim 3 , wherein the particular characteristic includes a time duration of at least one phase of the job, and wherein setting at least the particular characteristic of the job profile comprises varying the time duration of the at least one phase of the job by applying the at least one scaling factor to the size of the data to be processed.

5. The method of claim 4 , wherein the time duration of the at least one phase comprises a time duration of a shuffle phase and a time duration of a reduce phase, wherein the reduce tasks are performed in the shuffle phase and the reduce phase.

6. The method of claim 5 , wherein the size of the data to be processed comprises a size of the intermediate results to be processed by a reduce stage that includes the reduce tasks, and wherein setting at least the particular characteristic of the job profile comprises varying a time duration of a phase of the reduce stage by applying the at least one scaling factor to the size of the data to be processed.

7. The method of claim 1 , further comprising:

determining, based on the estimated performance parameter, whether a performance goal of the job will be satisfied.

8. The method of claim 7 , further comprising receiving an indication of the allocated amount of resources for the job, wherein the allocated amount of resources comprises an allocated number of map slots and number of reduce slots, wherein the map tasks are performed in the map slots, and the reduce tasks are performed in the reduce slots.

9. The method of claim 1 , wherein providing the performance model comprises providing the performance model having a lower bound and an upper bound of the performance parameter.

10. The method of claim 1 , wherein receiving the job profile including the characteristics of the job includes receiving the job profile including plural ones of: a minimum time duration of a map task, an average time duration of a map task, a maximum time duration of a map task, an average size of input data for a map task, an average time duration of a reduce task, and a maximum time duration of a reduce task.

11. An article comprising at least one non-transitory machine-readable storage medium storing instructions that upon execution cause a system having a processor to:

receive a job profile that includes characteristics of a job to be executed, wherein the characteristics of the job profile relate to map tasks and reduce tasks of the job, wherein the map tasks produce intermediate results based on input data, and the reduce tasks produce an output based on the intermediate results, and wherein the characteristics of the job profile include at least one particular characteristic that varies according to a size of data to be processed;

scale at least the particular attribute of the job profile based on the size of the data to be processed;

provide a performance model that calculates a performance parameter based on the characteristics of the job profile including the scaled particular attribute; and

determine, using a value of the performance parameter calculated by the performance model, an allocation of resources to assign to the job to meet a performance goal associated with the job.

12. The article of claim 11 , wherein the instructions upon execution cause the system to further:

compute at least one scaling factor according to the size of the data to be processed, wherein the scaling is performed using the at least one scaling factor.

13. The article of claim 12 , wherein computing the at least one scaling factor is based on solving for the at least one scaling factor based on measurements made in a number of experiments in which the job is executed with datasets of different sizes.

14. The article of claim 11 , wherein the performance goal is a completion time, and wherein the performance parameter is a time value.

15. The article of claim 11 , wherein scaling at least the particular attribute of the job profile comprises scaling a duration of a phase of the reduce tasks based on the size of the data to be processed.

16. The article of claim 15 , wherein scaling the duration of the phase of the reduce tasks comprises scaling the duration of a shuffle phase or a reduce phase of the reduce tasks.

17. The article of claim 12 , wherein scaling at least the particular attribute of the job profile comprises applying the at least one scaling factor that causes modification of a duration of a phase of the reduce tasks.

18. A system comprising:

a storage medium to store a job profile that includes characteristics of a job to be executed, wherein the characteristics of the job profile relate to map tasks and reduce tasks of the job, wherein the map tasks produce intermediate results based on input data, and the reduce tasks produce an output based on the intermediate results, and where the characteristics of the job profile include at least one particular characteristic that varies according to a size of data to be processed; and

at least one hardware processor to:

set at least the particular characteristic of the job profile based on the size of the data to be processed;

provide a performance model based on the job profile containing the set particular characteristic and an allocated amount of resources for the job; and

estimate a performance parameter of the job using the performance model.

19. The system of claim 18 , wherein the particular characteristic comprises a duration of a phase of the reduce tasks, and wherein the setting of at least the particular characteristic comprises modifying the duration of the phase of the reduce tasks based on the size of the data to be processed.

20. The system of claim 19 , wherein the at least one processor is to further compute at least one scaling factor based on the size of the data to be processed, wherein the modifying of the duration of the phase of the reduce tasks comprises computing the duration using the at least one scaling factor.