Systems and methods for autoscaling instance groups of computing platforms
Systems and methods scale an instance group of a computing platform by determining whether to scale up or down the instance group by using historical data from prior jobs wherein the historical data includes one or more of: a data set size used in a prior related job and a code version for a prior related job. The systems and methods also scale the instance group up or down based on the determination. In some examples, systems and methods scale an instance group of a computing platform by determining a job dependency tree for a plurality of related jobs, determining runtime data for each of the jobs in the dependency tree and scaling up or down the instance group based on the determined runtime data.
1. A method for scaling an instance group of a computing platform, the method comprising:
determining whether to scale up or down the instance group by using historical data from prior jobs wherein the historical data comprises one or more of: a data set size used in a prior related job and a code version for a prior related job; and
scaling the instance group up or down based on the determination;
wherein the determining whether to scale up or down the instance group comprises comparing a current job code version with a code version for a prior related job, and
wherein the method is performed using one or more processors.
2. The method of claim 1 wherein the determining whether to scale up or down the instance group comprises comparing a planned data set size to be used for a job with the data set size used in a prior related job.
3. The method of claim 1 wherein the determining whether to scale up or down the instance group comprises changing a weighting associated with the historical data based on whether a code version to run a prior job has changed.
4. A computer-implemented system for scaling an instance group of a computing platform, the system comprising:
one or more processors; and
a memory storing instructions, the instructions, when executed by the one or more processors, causing the system to perform:
determining whether to scale up or down the instance group by using historical data from prior jobs wherein the historical data comprises one or more of: a data set size used in a prior related job and a code version for a prior related job; and
scaling the instance group up or down based on the determination,
wherein the determining whether to scale up or down the instance group comprises comparing a current job code version with a code version for a prior related job.
5. The system of claim 4 wherein the memory stores instructions, the instructions, when executed by the one or more processors, causing the system to further compare a planned data set size to be used for a job with the data set size used in a prior related job.
6. The system of claim 4 , wherein the memory stores instructions, the instructions, when executed by the one or more processors, causing the system to further change a weighting associated with the historical data based on whether a code version to run a prior job has changed.
7. A computer-implemented system for scaling an instance group of a computing platform, the system comprising:
one or more processors; and
a memory storing instructions, the instructions, when executed by the one or more processors, causing the system to perform:
determining a job dependency tree for a plurality of related jobs,
determining runtime data for each of the jobs in the dependency tree; and
scaling up or down the instance group based on the determined runtime data.
8. The system of claim 7 wherein the memory stores instructions, the instructions, when executed by the one or more processors, causing the system to generate the dependency tree for the plurality of related jobs based on data sets that depend on each other such that a data set output from one job serves as an input data set for one or more other jobs.
9. The system of claim 7 wherein the memory stores instructions, the instructions, when executed by the one or more processors, causing the system to determine runtime data for each of the jobs in the dependency tree by determining an accumulated runtime length for all jobs in the dependency tree for a job.
10. The system of claim 7 wherein the memory stores instructions, the instructions, when executed by the one or more processors, causing the system to:
determine whether to scale up or down the instance group by using historical data from prior jobs wherein the historical data comprises one or more of: a data set size used in a prior related job and a code version for a prior related job; and
scale the instance group up or down based on the determination.
11. The system of claim 10 wherein the memory stores instructions, the instructions, when executed by the one or more processors, causing the system to compare a planned data set size to be used for a job with the data set size used in a prior related job.
12. The system of claim 10 wherein the memory stores instructions, the instructions, when executed by the one or more processors, causing the system to the determine whether to scale up or down the instance group by comparing a current job code version with a code version for a prior related job.