SYSTEM TO OPTIMIZE THE INSTANCE SIZE AND CLUSTER SIZE FOR JOBS RUNNING ON DISTRIBUTED COMPUTING CLUSTERS
A method for optimizing a workflow provided to a cloud computing system is described. The method includes extracting information from at least one log file for a job. The log file(s) are for at least one run of the job (e.g., correspond to a run of the job). The method also includes determining a recommended allocation of cloud resources for the job based on the information from the log file(s). The recommended allocation of the cloud resources includes an incremental change from a most recent allocation of cloud resources.
1 . A method for optimizing a workflow provided by a computing platform to a cloud computing system, comprising:
extracting information from at least one log file for a job, the at least one log file for at least one run of the job; and
determining a recommended allocation of cloud resources for the job based on the information from the at least one log file, the recommended allocation of cloud resources including an incremental change from a most recent allocation of cloud resources.
2 . The method of claim 1 , wherein the incremental change comprises a change of at most ten percent or one unit of measurement for at least one of the cloud resources.
3 . The method of claim 1 , wherein the determining the recommended allocation further includes:
mapping the information to an intermediate fundamentals domain.
4 . The method of claim 3 , wherein the intermediate fundamentals domain includes at least one of a storage bandwidth, a network bandwidth, a CPU architecture, a clock rate, a virtual CPU, or a memory.
5 . The method of claim 1 , wherein the determining the recommended allocation further includes:
mapping the information to an optimization domain including at least one of a worker number, a worker virtual CPU number, a worker network bandwidth, a worker storage bandwidth, a worker clock rate, or a memory per worker.
6 . The method of claim 5 , wherein the determining the recommended allocation further includes:
providing the recommended allocation based on at least one of a cost, a run time, or a periodicity of the job.
7 . The method of claim 6 , wherein the run time is based on at least one of serial computation process costs, parallel computation process costs, inter-worker communication costs, network bandwidth runtime contributions, or periodic runtime variations of the job.
8 . The method of claim 1 , wherein determining the recommended allocation includes determining a sequence of allocations for the job and the recommended allocation is a first allocation of the sequence of allocations.
9 . The method of claim 8 , wherein an allocation of the sequence of allocations includes an incremental change from a previous allocation of the sequence of allocations.
10 . A system, comprising:
a processor configured to:
extract information from at least one log file for a job, the at least one log file for at least one run of the job; and
determine a recommended allocation of cloud resources for the job based on the information from the at least one log file, the recommended allocation of cloud resources including an incremental change from a most recent allocation of cloud resources; and
a memory coupled to the processor and configured to provide the processor with instructions.
11 . The system of claim 10 , wherein the incremental change comprises a change of at most ten percent or one unit of measurement for at least one of the cloud resources.
12 . The system of claim 10 , wherein the processor is further configured to provide the recommended allocation based on at least one of a cost, a run time, or a periodicity of the job.
13 . The system of claim 12 , wherein the run time is based on at least one of serial computation process costs, parallel computation process costs, inter-worker communication costs, network bandwidth runtime contributions, or periodic runtime variations of the job.
14 . The system of claim 10 , wherein determining the recommended allocation includes determining a sequence of allocations for the job and the recommended allocation is a first allocation of the sequence of allocations.
15 . The system of claim 14 , wherein an allocation of the sequence of allocations includes an incremental change from a previous allocation of the sequence of allocations.
16 . A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:
extracting information from at least one log file for a job, the at least one log file for at least one run of the job; and
determining a recommended allocation of cloud resources for the job based on the information from the at least one log file, the recommended allocation of cloud resources including an incremental change from a most recent allocation of cloud resources.
17 . The computer program product of claim 16 , wherein the incremental change comprises a change of at most ten percent or one unit of measurement for at least one of the cloud resources.
18 . The computer program product of claim 16 , wherein the computer instructions further include computer instructions for:
providing the recommended allocation based on at least one of a cost, a run time, or a periodicity of the job.
19 . The computer program product of claim 18 , wherein the run time is based on at least one of serial computation process costs, parallel computation process costs, inter-worker communication costs, network bandwidth runtime contributions, or periodic runtime variations of the job.
20 . The computer program product of claim 16 , wherein determining the recommended allocation includes determining a sequence of allocations for the job and the recommended allocation is a first allocation of the sequence of allocations.