Heterogeneous auto-scaling big-data clusters in the cloud
The present invention is generally directed to systems and methods of provisioning, and using heterogeneous clusters in a cloud-based big data system, the heterogeneous clusters made up of primary instance types and different types of instances, the method including: determining if there are composition requirements of any heterogeneous cluster, the composition requirements defining instance types permitted for use; determining if any of the permitted different types of instances are required or advantageous for use; determining an amount of different types of instances to utilize, this determination based at least in part on an instance weight; provisioning the heterogeneous cluster comprising both primary instances and permitted different types of instances.
1. A method for heterogeneously auto-scaling cloud-based big data clusters to use multiple instance types, comprising:
determining a primary instance type, the primary instance type selected from a group of approved instances with varying amounts of diskspace and/or computer processing units (CPUs);
determining provisioning requirements based on characteristics of the primary instance type including associated amount of diskspace and/or CPUs;
assigning a weight of 1.0 to the primary instance type;
identifying other approved instance types and determining an instance weight for each other approved instance type, wherein the determination of instance weight for each other approved instance type is based upon a comparison with the characteristics of the primary instance;
determining if using any other instance types is advantageous in that it would result in reduced costs or faster processing;
if using other instance types is advantageous, determining which other instance types to use, and determining the number of other instance types by using the instance weight to determine corresponding processing power.
2. The method of claim 1 , wherein the step of determining an instance weight for each other approved instance type further comprises:
determining the disk space of each instance type;
determining a ratio of the disk space of each instance type compared with the primary instance type;
deriving an instance weight for each instance type based on its ratio of disk space compared to the primary instance type.
3. The method of claim 1 , wherein the step of determining an instance weight for each other approved instance type further comprises:
identifying the number of computer processing units (CPUs) in each other approved instance type;
identifying the number of CPUs in the primary instance type;
calculating a ratio for each other approved instance type between the number of CPUs in each other approved instance type and the number of CPUs in the primary instance type;
deriving an instance weight for each other instance type based on the ratio determined for each other instance type.