Cloud to bare metal correlation for performance estimation and prediction
Systems and methods for correlating performance on bare metal systems with virtualized instances such as those commonly used in cloud computing systems are disclosed. Performance data may be collected from different bare metal and cloud instances. The performance data may be used to predict the performance of an application on another system, even if the particular performance counter of interest is unavailable on the system. Using measured and estimated performance counters (e.g., instructions counters), multiple measured and unmeasured but predicted instances can be compared and sorted to assist the user in making an informed decision when selecting where to run their instance and what configuration to use.
1 . A method for estimating performance on cloud computing systems, the method comprising:
executing a plurality of performance benchmarks on a plurality of cloud computing systems and bare metal computing systems to collect performance counter data, wherein the performance counter data comprises a set of available counters and one or more unavailable counters, wherein the one or more unavailable counters is a restricted-access performance counter;
creating a model for each of the plurality of performance benchmarks by identifying a statistical or machine-learned correlation between the set of available counters and the one or more unavailable counters using the performance counter data collected from execution of the plurality of performance benchmarks on the plurality of cloud computing systems and bare metal computing systems;
matching an application to be modeled with one of the plurality of performance benchmarks, in response to receiving the performance counter data for the application;
predicting at least one unavailable counter of the one or more unavailable counters for the application based on the model created for the matched performance benchmark, in response to matching the application with the performance benchmark;
deploying a recommended cloud computing system configuration for the application and executing the application, the recommended cloud computing system based on the predicted at least one unavailable counter to optimize application performance or cost, in response to predicting the at least one unavailable counter of the one or more unavailable counters; and
wherein the model created for the matched performance benchmark is configured to receive, as input, the performance counter data for the application collected from a cloud computing system, and to output the predicted at least one unavailable counter based on the statistical or machine-learned correlations derived from prior benchmark executions including both the cloud and the bare metal computing systems.
2 . The method of claim 1 , further comprising populating a matrix based on the collected performance counter data, wherein the model for each of the plurality of performance benchmarks is a vector.
3 . The method of claim 2 , further comprising updating the matrix with additional data gathered from executing applications on one or more of the plurality of cloud computing systems.
4 . The method of claim 2 , wherein the matrix has columns, wherein one or more of the columns of the matrix stores values for performance counters comprising cycles, page-faults, and context-switches.
5 . The method of claim 1 , wherein the matching is performed using normalized histograms of performance counters.
6 . The method of claim 1 , wherein the model for each of the plurality of performance benchmarks is a regression model based on linear regression with a positive coefficients constraint.
7 . The method of claim 1 , wherein the one or more unavailable counters comprise an instructions counter.
8 . The method of claim 1 , wherein predicting at least one unavailable counter is based on a positive and linear relationship between the set of available counters and the one or more unavailable counters.
9 . The method of claim 1 , further comprising calculating a predicted relative performance between different cloud instances on different clouds for the application.
10 . A system having at least one processor, the system configured to:
execute a plurality of performance benchmarks on a plurality of cloud computing systems and bare metal computing systems to collect performance counter data, wherein the performance counter data comprises a set of available counters and one or more unavailable counters, wherein the one or more unavailable counters is a restricted-access performance counter;
create a model for each of the plurality of performance benchmarks by identifying a statistical or machine-learned correlation between the set of available counters and the one or more unavailable counters in the collected performance counter data, in response to determining that the performance counter data for both the available counters and the one or more unavailable counters has been collected for that performance benchmark;
match an application to be modeled with one of the plurality of performance benchmarks, in response to receiving the performance counter data for the application;
predict at least one unavailable counter of the one or more unavailable counters for the application based on the model created for the matched performance benchmark, in response to matching the application with the performance benchmark;
deploy a recommended cloud computing system configuration for the application and execute the application, the recommended cloud computing system configuration based on the predicted at least one unavailable counter to optimize application performance or cost, in response to predicting the at least one unavailable counter of the one or more unavailable counters; and
wherein the model created for the matched performance benchmark is configured to receive, as input, the performance counter data for the application collected from a cloud computing system, and to output the predicted at least one unavailable counter based on the statistical or machine-learned correlations derived from prior benchmark executions including both the cloud and the bare metal computing systems.
11 . The system of claim 10 , further configured to populate a matrix based on the collected performance counter data, wherein the model for each of the plurality of performance benchmarks is a vector.
12 . The system of claim 11 , further configured to update the matrix with additional data gathered from executing applications on one or more of the plurality of cloud computing systems.
13 . The system of claim 11 , wherein the matrix has columns, wherein one or more of the columns of the matrix stores values for performance counters comprising cycles, page-faults, and context-switches.
14 . The system of claim 10 , wherein the match is performed using normalized histograms of performance counters.
15 . The system of claim 10 , wherein the model for each of the plurality of performance benchmarks is a regression model based on linear regression with a positive coefficients constraint.
16 . The system of claim 10 , wherein the one or more unavailable counters comprise an instructions counter.
17 . A system for estimating application performance comprising:
a memory including computer readable instructions;
at least one processor configured to execute the computer readable instructions to:
collect performance counter data from test runs of a plurality of performance benchmarks on a plurality of cloud computing systems, wherein the performance counter data comprises a set of available counters and one or more unavailable counters, wherein the one or more unavailable counters is a restricted-access performance counter;
create a model for each of the plurality of performance benchmarks based on an observed relationship between the set of available counters and the one or more unavailable counters in the collected performance counter data, in response to determining that the performance counter data for both the available counters and the one or more unavailable counters has been collected for that performance benchmark;
match an application to be modeled with one of the plurality of performance benchmarks, in response to receiving the performance counter data for the application;
predict at least one unavailable counter of the one or more unavailable counters for the application based on the model created for the matched performance benchmark's model, in response to matching the application with the performance benchmark;
deploy, in response to predicting the at least one unavailable counter of the one or more unavailable counters, a recommended cloud computing system configuration for the application and execute the application, the recommended cloud computing system configuration corresponds to a configuration that satisfies one or more optimization criteria including performance, cost, or equivalence to a reference computing system based on a comparison of the predicted performance across a plurality of cloud computing system configurations.
18 . The system of claim 17 , wherein the at least one processor is further configured to use normalized histograms of the performance counter data to match the application to be modeled with one of the plurality of performance benchmarks.
19 . The system of claim 17 , wherein the model for each of the plurality of performance benchmarks is a regression model based on linear regression with a positive coefficients constraint.
20 . The system of claim 17 , wherein the one or more unavailable counters comprise an instructions counter.
21 . A method for comparing performance of bare metal and non-bare metal systems, the method comprising:
building a training set by gathering performance data by running one or more performance benchmarks on multiple bare metal and cloud instances that permit access to available performance counters;
creating a regression model trained with the gathered performance data to learn a mapping between the available performance counters and one or more unavailable performance counters, in response to determining that the training set includes performance data for both the available performance counters and the one or more unavailable counters, wherein the one or more unavailable counters is a restricted-access performance counter;
inferring, using the regression model, an unavailable performance counter of the one or more unavailable counters for a new cloud instance, in response to receiving performance counter data for the new cloud instance;
comparing, in response to inferring the unavailable performance counter for the new cloud instance, an inferred unavailable performance counter value for the new cloud instance with corresponding measured values obtained from prior executions of the one or more performance benchmarks on the bare metal instances to estimate a relative performance difference between the cloud instances and bare metal configurations of the bare metal instances for the same benchmark or application;
recommending, in response to and based on the estimated relative performance difference, a cloud configuration that approximates the performance of a corresponding bare metal configuration or optimizes the cloud configuration for a selected performance metric; and
deploying the recommended cloud configuration and executing the application thereon.