Cost-based optimization of configuration parameters and cluster sizing for hadoop
Cost-based optimization of configuration parameters and cluster sizing for distributed data processing systems are disclosed. According to an aspect, a method includes receiving at least one job profile of a MapReduce job. The method also includes using the at least one job profile to predict execution of the MapReduce job within a plurality of different predetermined settings of a distributed data processing system. Further, the method includes determining one of the predetermined settings that optimizes performance of the MapReduce job. The method may also include automatically adjusting the distributed data processing system to the determined predetermined setting.
1. A method comprising:
at a processor and memory:
collecting monitoring data during execution of a MapReduce job comprising a first set of configuration settings on a MapReduce framework,
wherein collecting the monitoring data comprises applying dynamic instrumentation to the MapReduce job and the MapReduce framework during execution of the MapReduce job on the MapReduce framework,
wherein applying dynamic instrumentation comprises applying a specified set of event-condition-action (ECA) rules to the MapReduce framework, and
wherein an event comprises at least one of a entry function, exit function, memory allocation, system call event occurring during execution of the MapReduce job on the MapReduce framework;
generating dataflow fields and cost fields of a job profile of the MapReduce job based on the collected monitoring data; and
communicating the job profile to a prediction process configured to predict the behavior of the MapReduce job comprising a second set of configuration settings on the MapReduce framework using the job profile.
2. The method of claim 1 , wherein collecting monitoring data comprises receiving one of a dataflow measure, an execution time measure, and a resource usage measure during execution of the MapReduce job on the MapReduce framework.
3. The method of claim 1 , wherein collecting monitoring data comprises receiving run-time monitoring information from the MapReduce job and MapReduce framework during execution of the MapReduce job on the MapReduce framework.
4. The method of claim 1 , wherein the data flow fields of the job profile comprises dataflow information associated with the collected monitoring data of the MapReduce job.
5. The method of claim 4 , wherein the dataflow information comprises one of a size of data and I/O transfer processed during execution of the MapReduce job.
6. The method of claim 1 , wherein the cost fields of the job profile comprise cost information associated with the collected monitoring data of the MapReduce job.
7. The method of claim 6 , wherein the cost information comprises one of resource usage and execution time associated with the collected monitoring data of the MapReduce job.
8. The method of claim 1 , wherein the job profile fields comprises one of a dataflow statistics field and a cost statistic field.
9. The method of claim 1 , wherein using the job profile comprises using the job profile to simulate execution of the MapReduce job on the MapReduce framework, the second set of configuration settings of the MapReduce job being different from the first set of configuration settings used to collect the monitoring data.
10. The method of claim 1 , wherein the MapReduce job comprises map tasks and reduce tasks.
11. The method of claim 1 , wherein an action comprises at least one of obtaining the duration of a function call, examining the memory state, and counting the number of bytes transferred during execution of the MapReduce job on the MapReduce framework.
12. The method of claim 1 , wherein at least one of the dataflow fields comprises the amount of data flowing through at least one task phase during the execution of the MapReduce job on the MapReduce framework.
13. A computing device comprising:
a processor and memory configured to:
collect monitoring data during execution of a MapReduce job comprising a first set of configuration settings on the MapReduce framework,
wherein collecting the monitoring data comprises applying dynamic instrumentation to the MapReduce job and the MapReduce framework during execution of the MapReduce job on the MapReduce framework,
wherein applying dynamic instrumentation comprises applying a specified set of event-condition-action (ECA) rules to the MapReduce framework, and
wherein an event comprises at least one of a entry function, exit function, memory allocation, system call event occurring during execution of the MapReduce job on the MapReduce framework;
generate dataflow fields and cost fields of a job profile based on the collected monitoring data; and
communicate the job profile to a prediction process configured to predict the behavior of the MapReduce job comprising a second set of configuration settings on the MapReduce framework using the job profile.