Heterogeneous auto-scaling big-data clusters in the cloud

1. A method of provisioning and using heterogeneous clusters in a cloud-based big data system, the heterogeneous clusters comprised of primary instance types and different types of instances, the method comprising:

determining if there are composition requirements of any heterogeneous cluster, the composition requirements defining instance types permitted for use;

determining if any of the permitted different types of instances are required or advantageous for use;

determining an amount of different types of instances to utilize, this determination based at least in part on an instance weight;

provisioning the heterogeneous cluster comprising both primary instances and permitted different types of instances.

2. The method of claim 1 , wherein different types of instances are required when there are not enough primary instances available to perform a specific task.

3. The method of claim 1 , wherein different types of instances are advantageous when different types of instances have reduced costs.

4. The method of claim 3 , wherein different types of instances comprise spot instances.

5. The method of claim 1 , wherein determining instance comprises:

identifying the number of computer processing units (CPUs) in each other approved instance type;

identifying the number of CPUs in the primary instance type;

calculating a ratio for each other approved instance type between the number of CPUs in each other approved instance type and the number of CPUs in the primary instance type;

deriving an instance weight for each other instance type based on the ratio determined for each other instance type.

6. The method of claim 1 , further comprising running workloads across various instance types.