Optimizing execution and resource usage in large scale computing

1. A method for tuning workflow settings in a distributed computing workflow comprising sequential interdependent jobs, the method comprising:

pairing a terminal stage of a first job and a leading stage of a second, sequential job to form an optimization pair, in which data segments output by the terminal stage of the first job comprises data input for the leading stage of the second job;

tuning a performance of the optimization pair by:

determining, with a computational processor, an estimated minimum execution time for the optimization pair;

increasing the minimum execution time to generate an increased execution time; and

calculating, with the computational processor, a minimum number of data segments produced by the terminal stage that still permit execution of the optimization pair within the increased execution time; and

executing, by distributed computing devices, the optimization pair to produce the minimum number of data segments.

2. The method of claim 1 , in which the first job and the second jobs are distributed computing jobs performed by a plurality of distributed computing devices.

3. The method of claim 1 , in which the distributed computing workflow comprises a MapReduce workflow comprising a series of sequential MapReduce jobs executed on a distributed computing system, the terminal stage of the first job comprises a reduce stage, and the leading stage of the second job comprises a map stage.

4. The method of claim 1 , further comprising dividing input data to be processed into the minimum number of data segments, in which the amount of input data to be processed and the minimum number of data segments that the input data is divided into determines the amount of data contained in each of the data segments, in which the data segments are distributed among the distributed computing devices for parallel operations.

5. The method of claim 1 , in which calculating the minimum execution time comprises:

performing microbenchmarks on the distributed computing devices to create a platform model that quantifies relationships between an amount of data processed and completion time for each stage;

given the input data set and a baseline number of data segments, determining an amount of data in each data segment; and

using a platform model to calculate the estimated completion time for each stage.

6. The method of claim 1 , further comprising global tuning of the distributed computing workflow by:

pairing each group of interdependent stages in the workflow to produce a plurality of optimization pairs;

calculating a reduction in resource usage for each optimization pair produced by execution of the optimization pair within the increased execution time; and

selecting an optimization pair with the highest reduction of resource usage to execute with the increased execution time.

7. The method of claim 6 , further comprising calculating an estimated execution time for the workflow by:

calculating the estimated execution time for each stage in each job;

summing the estimated execution time for each stage in each job to obtain the estimated execution time for each job; and

summing the estimated execution times for each job to produce the estimated execution time for the workflow.

8. The method of claim 1 , further comprising generating a model ensemble to predict performance of the distributed computing work flow on the distributed computing devices and to inform trade-offs between execution time and resource usage, in which the model ensemble comprises:

a platform model describing a relationship between an amount of input data and completion time for each stage;

workflow settings defining a number of reduce tasks for each job; and

a workflow model to calculate an amount of data flowing through the jobs, given the workflow settings.

9. The method of claim 8 , in which estimated completion times of the stages are calculated by inputting the amount of data flowing through the stage into the platform model.