Performance benchmarking and characterization for cloud and bare metal systems
A system and method for benchmarking the performance of computing resources in a distributed computing system is disclosed. The system may comprise a performance database, a training program, and a machine learning engine. The training program may be configured to create a training data set by running a plurality of workloads multiple times on a plurality of different cloud computing systems using a plurality of different configurations. The training data may be stored in the performance database, and the machine learning engine may be configured to use the training data to create a model usable to predict the performance of future jobs on different cloud computing systems and bare metal systems.
1 . A method for predicting performance of a job, the method comprising:
creating a training data set by running different test applications on different cloud computing systems using different configurations, wherein the different test applications are selected to test specific performance characteristics of the cloud computing systems and configurations, and the training data set includes performance feature vectors associated with individual test applications and configurations;
training a machine learning engine using the training data set, the training incorporating performance feature data extracted from the different test applications and associating the performance feature data with specific configurations;
predicting a performance level for a first future job using the machine learning engine, wherein the prediction comprises comparing one or more performance characteristics of the first future job against corresponding performance characteristics of previously executed test applications stored in a performance database;
generating a ranked list of computing system configurations based on the predicted performance level for the first future job, wherein the ranked list is ordered according to one or more performance-limiting factors identified for the first future job, including at least one of CPU utilization, memory bandwidth, GPU capability, or I/O throughput;
generating, using the ranked list, one or more recommendations of computing system configurations for the first future job;
configuring and deploying the first future job on a computing system in accordance with a computing system configuration selected from the one or more recommendations;
calculating a level of confidence for the predicted performance level;
detecting whether there is insufficient data present in the training data set to achieve a predetermined level of confidence in the predicting; and
in response to insufficient data being present, creating a list of additional test application and cloud computing system combinations for testing that will enable achieving the predetermined level of confidence.
2 . The method of claim 1 , further comprising:
using heuristics to fill in missing data; and
performing cross-validation of the machine learning engine.
3 . The method of claim 1 , further comprising:
collecting job metadata; and
using the job metadata in the predicting.
4 . The method of claim 1 , further comprising displaying a generic performance ranking based on CPU in response to the predicted performance level for the first future job being below a predetermined confidence level.
5 . The method of claim 3 , further comprising:
running a test of the first future job on a first computing system; and
capturing a set of performance data for the test, wherein the predicting comprises
using the set of performance data in lieu of the job metadata.
6 . The method of claim 1 , further comprising predicting an eigen feature for the first future job for each of the different cloud computing systems.
7 . The method of claim 1 , further comprising calculating an eigen feature for each test application in the training data for each of the different cloud computing systems.
8 . The method of claim 1 , further comprising predicting which of the different cloud computing systems is likely to have a largest variance in absolute performance across multiple runs of the first future job.
9 . A non-transitory computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the steps of the method of claim 1 .
10 . A system for managing computing jobs, the system comprising:
a performance database;
at least one processor;
a training program configured to create a training data set by running different test applications on different cloud computing systems using different configurations, wherein the training program is configured to store the training data set in the performance database, and the different test applications are selected to test specific performance characteristics of the cloud computing systems and configurations, and the training data set includes performance feature vectors associated with individual test applications and configurations;
a machine learning engine configured to read the training data set from the performance database and predict a performance level for a first future job based on the training data set, wherein the prediction comprises comparing one or more performance characteristics of the first future job against corresponding performance characteristics of previously executed test applications stored in the performance database;
a ranking module configured to generate a performance ranking based on CPU in response to a predicted performance level of the first future job being below a predetermined minimum confidence level;
wherein the machine learning engine is configured to generate a ranked list of computing system configurations based on the predicted performance level for the first future job and generate, using the ranked list, one or more recommendations of computing system configurations for the first future job;
wherein the machine learning engine is further configured to detect whether there is sufficient data present in the performance database to achieve the predetermined minimum confidence level in performance prediction, and in response to insufficient data being present, cause the training program to execute additional tests on additional cloud computing system combinations to achieve the predetermined minimum confidence level;
wherein the ranked list is ordered according to one or more performance-limiting factors identified for the first future job, including at least one of CPU utilization, memory bandwidth, GPU capability, or I/O throughput; and
wherein the at least one processor is configured to configure and deploy the first future job on a computing system in accordance with a computing system configuration selected from the one or more recommendations.
11 . The system of claim 10 , wherein the machine learning engine uses a K-Nearest Neighbor algorithm.
12 . The system of claim 10 , further comprising:
a user interface configured to prompt a user to enter job metadata for the first future job, wherein the machine learning engine is configured to use the job metadata in predicting the performance level of the first future job.
13 . The system of claim 10 , further comprising:
a performance measurer configured to execute a test of a second future job on a first cloud computing system, and
a performance data collector configured to capture a set of test performance data for the test and
store the set of test performance data in the performance database, wherein the machine learning engine is configured to use the test performance data to predict the performance level of the second future job.
14 . The system of claim 10 , wherein the machine learning engine is further configured to determine an eigen feature for a combination of a particular cloud system and the first future job.
15 . The system of claim 10 , wherein the machine learning engine is further configured to perform a sensitivity analysis and find derivatives and monotonic groups within sets of applications or sets of computing system-configuration combinations.
16 . The system of claim 10 , wherein the different configurations includes one or more of a number and/or type of central processing units (CPUs) and a number and/or type of graphics processing units (GPUs).
17 . The system of claim 10 , wherein the different configurations includes one or more of CPU-memory bandwidth, GPU-memory bandwidth, input/output (IO) options, and latency.
18 . The system of claim 10 , wherein the machine learning engine uses supervised or unsupervised learning.
19 . The method of claim 1 further comprising selecting different test applications to test different aspects of performance including central processing unit (CPU) performance, graphics processing unit (GPU) performance, CPU-memory bandwidth, GPU-memory bandwidth, IO bandwidth, and latency, wherein the selected different test applications are each run on the different cloud computing systems using the different configurations to generate a matrix of performance data to train the machine learning engine.
20 . The method of claim 1 wherein each configuration has different performance features that result in different performance characteristics, and wherein the performance characteristics include one or more of a) identification of hot blocks, code bottlenecks, and resource intensive portions of the application, b) instructions per second (IPS) or run-time, c) performance logs or tool output, and d) scoring predefined performance vectors.