Best trade-off point on an elbow curve for optimal resource provisioning and performance efficiency
View Patent ↗A method, procedure and algorithm are provided for efficient resource provisioning in Hadoop MapReduce. The crux of the method, procedure and algorithm is not tied to any specific system, and can be applied to many processes and devices. It provides a general approach and techniques based on an algorithm with mathematical formulas to find the Best Trade-off Point on an elbow curve, non-inverted or inverted, of performance vs. resources. It is applicable to any systems relying on a trade-off elbow curve for making good decision.
1. A method, algorithm or process for calculating an exact optimal number of resources for a job that provides the best trade-off point between performance vs. resources on an elbow curve ƒ(x)=(a/x)+b fitted from sampled executions of the target system, comprising the steps of:
(A) providing a completed configuration and fine tuning of the architecture, software and hardware of a production system targeted for calibration;
(B) collecting necessary preview job performance data from historical runtime performances or sampled executions on said the same targeted production system, configured exactly as in step A, as reference points for each workload;
(C) curve-fitting said preview job performance data to obtain fit parameters a and b in an elbow curve function ƒ(x)=(a/x)+b, where x is the number of resources;
(D) inputting said fit parameter a to a Best-Trade-off-Point algorithm to obtain a recommended optimal number of resources for a workload as output, wherein said Best-Trade-off-Point algorithm includes the steps of: computing the number of resources over a range of slopes from the first derivative off(x)=(a/x)+b and the acceleration over a range of slopes from the second derivative of ƒ(x)=(a/x)+b; applying the Chain rule to search for break points and major plateaus on the graphs of acceleration, slope, and task resources over a range of incremental changes in acceleration per slope increment; and extracting the exact number of resources at the best trade-off point on the elbow curve and outputs it as recommended optimal number of resources for a workload;
(E) repeating steps (B)-(D) gather sufficient resource provisioning data points for different workloads to build a database of resource consumption signatures for subsequent job profiling;
(F) repeating steps (A)-(E) to recalibrate said database of resource consumption signatures if there are any major changes to step A; and
(G) using said database of resource consumption signatures to match dynamically submitted production jobs to their recommended optimal number of resources for efficient resource provisioning.
2. A method, algorithm or process for calculating an exact optimal number of resources for a job that provides the best trade-off point between performance vs. resources on an inverted elbow curve ƒ(x)=−(a/x)+b fitted from sampled executions of the target system, comprising the steps of:
(A) providing a completed configuration and fine tuning of the architecture, software and hardware of a production system targeted for calibration;
(B) collecting necessary preview job performance data from historical runtime performances or sampled executions on said the same targeted production system, configured exactly as in step A, as reference points for each workload;
(C) curve-fitting said preview job performance data to obtain fit parameters a and b in an inverted elbow curve function ƒ(x)=−(a/x)+b, where x is the number of resources;
(D) inputting said fit parameter a to a Best-Trade-off-Point algorithm to obtain a recommended optimal number of resources for a workload as output, wherein said Best-Trade-off-Point algorithm includes the steps of: computing the number of resources over a range of slopes from the first derivative of ƒ(x)=−(a/x)+b and the acceleration over a range of slopes from the second derivative of ƒ(x)=−(a/x)+b; applying the Chain rule to search for break points and major plateaus on the graphs of acceleration, slope, and task resources over a range of incremental changes in acceleration per slope increment; and extracting the exact number of resources at the best trade-off point on the inverted elbow curve and outputs it as recommended optimal number of resources for a workload;
(E) repeating steps (B)-(D) gather sufficient resource provisioning data points for different workloads to build a database of resource consumption signatures for subsequent job profiling;
(F) repeating steps (A)-(E) to recalibrate said database of resource consumption signatures if there are any major changes to step A; and
(G) using said database of resource consumption signatures to match dynamically submitted production jobs to their recommended optimal number of resources for efficient resource provisioning.