Dynamic resource allocation based on fingerprint extraction of workload telemetry data
Resource allocation to workloads is disclosed. Telemetry data associated with existing or previously executed workloads is stored and used to develop models. Telemetry data from new workloads are collected and, using the models, a fingerprint is extracted and compared to the fingerprints of previous workloads. This allows the initial allocation of resources to the new workload to be improved and aids in resource allocation convergence.
1. A method for allocating resources, the method comprising:
generating a fingerprint from telemetry data associated with an execution for a new workload using an autoencoder;
comparing the fingerprint of the new workload to fingerprints associated with previously executed workloads;
identifying a closest match between the fingerprint of the new workload and a fingerprint of a previous workload from the comparisons;
allocating resources to the new workload that correspond to resources allocated to the previous workload; and
converging the resources allocated to the new workload and resources allocated to running workloads based on an error metric for the new workload and an error metric for each of the running workloads, wherein converging includes adjusting the resources allocated to the new workload and the resources allocated to the running workloads based on the error metrics.
2. The method of claim 1 , wherein the allocation of resources to the new workload is an initial allocation of the resources.
3. The method of claim 2 , further comprising converging the resources of the new workload and other running workloads such that the resources allocated to the new workload allow compliance with a services level agreement.
4. The method of claim 1 , wherein the telemetry data used in generating the fingerprint of the new workload is generated from an initial portion of the telemetry data that corresponds to a period of time at a beginning of an execution of the new workload.
5. The method of claim 4 , wherein the period of time is predetermined.
6. The method of claim 1 , wherein the autoencoder has been trained using telemetry data from the previous workloads.
7. The method of claim 1 , further comprising generating a new autoencoder based on the new workload when no match to the fingerprint of the new workload is found.
8. The method of claim 1 , further comprising collecting telemetry data from a plurality of workloads.
9. The method of claim 8 , further comprising training a plurality of models corresponding to a plurality of workloads and generating a plurality of fingerprints for each of the plurality of workloads.
10. The method of claim 1 , wherein the telemetry data includes at least one of CPU usage, memory usage, and/or network usage.
11. The method of claim 10 , further comprising generating time series data for the telemetry data.
12. The method of claim 1 , further comprising adjusting the resources allocated to the new workload based on the allocated resources, interference from other workloads, and a metric of interest.
13. The method of claim 1 further comprising allocating resources to comply with a given SLA and account for features of the new workload, and an infrastructure in which the new workload and other workloads concurrently run.
14. The method of claim 1 , wherein the autoencoder is trained by regressing data against itself such that an output of the autoencoder is similar to an input to the autoencoder.
15. The method of claim 1 , wherein comparing is performed based on features of the fingerprint and a distance between features of the fingerprint to features of the fingerprints from the previous workloads, wherein a match is determined when the distance is within a threshold distance.
16. The method of claim 1 , further comprising training an autoencoder per class of workload.
17. The method of claim 15 , further comprising: collecting telemetry data from workloads; generating time series of telemetry data; dividing the time series of telemetry data into windows; and generating fingerprints for each of the workloads from each of the windows.
18. The method of claim 1 , further comprising converging the workloads, wherein workloads that have converged are monitored and when an error of monitored workloads exceeds a threshold, the monitored workloads are actively controlled to converge the workloads.
19. A non-transitory computer readable storage comprising executable instructions for performing operations comprising:
generating a fingerprint from telemetry data associated with an execution for a new workload using an autoencoder;
comparing the fingerprint of the new workload to fingerprints associated with previously executed workloads;
identifying a closest match between the fingerprint of the new workload and a fingerprint of a previous workload from the comparisons;
allocating resources to the new workload that correspond to resources allocated to the previous workload;
allocating resources to the new workload that correspond to resources allocated to the previous workload; and
converging the resources allocated to the new workload and resources allocated to running workloads based on an error metric for the new workload and an error metric for each of the running workloads, wherein converging includes adjusting the resources allocated to the new workload and the resources allocated to the running workloads based on the error metrics.
20. The non-transitory computer readable storage of claim 19 , the operations further comprising converging the resources for the new workload and other workloads by considering an SLA, an infrastructure, interference, and allocated resources to the new workload.