LOW OVERHEAD PERFORMANCE DATA COLLECTION
Systems and methods for collecting performance data in high performance computing systems are disclosed. To prevent massive amounts of performance data from overwhelming the system and negatively impacting performance, collected performance data may be processed into two databases: (i) an aggregate database, and (ii) a time-series database holding the newest information for real time performance analysis. Storage space may be saved by using a FIFO buffer to store collected performance data. A real-time performance collection engine may adjust the performance sampling interval used and the particular performance counters used based on measured system impact and feedback from other system modules consuming the performance data.
1 . A method for managing performance data, the method comprising:
executing a performance profiling tool on a computing system;
executing an application on the computing system;
collecting performance data about the application from the performance profiling tool;
storing the performance data in a database;
monitoring an impact of the performance profiling tool on the application; and
adjusting an interval at which the performance profiling tool operates in order to keep the impact of the performance profiling tool on the application below a predetermined policy threshold.
2 . The method of claim 1 , further comprising:
providing the performance data to one or more system monitors;
receiving feedback from the system monitors; and
adjusting the interval at which the performance profiling tool operates based on the feedback.
3 . The method of claim 1 , further comprising:
providing the performance data to one or more system monitors;
receiving feedback from the system monitors; and
adding or removing one or more performance measures from being collected based on the feedback.
4 . The method of claim 2 , wherein one of the system monitors is a system health monitor.
5 . The method of claim 2 , wherein one of the system monitors is a system security monitor.
6 . The method of claim 2 , further comprising providing recommendations to a user regarding application optimizations based on the performance data stored in the database.
7 . The method of claim 2 , further comprising aggregating the performance data in the database.
8 . A method for estimating performance on cloud computing systems, the method comprising:
executing a plurality of performance benchmarks on a plurality of cloud computing systems and bare metal computing systems to collect performance counter data,
storing the performance counter data into a FIFO buffer;
reading the performance counter data out of the FIFO buffer;
storing a time-limited window of the performance counter data into a time-series database;
aggregating the performance counter data; and
storing the aggregated performance counter data into an aggregated database.
9 . The method of claim 8 , further comprising performing real-time performance analysis on the performance counter data in the time-series database and the aggregated database.
10 . The method of claim 8 , further comprising making recommendations to a user based on the performance counter data in the time-series database and the aggregated database.
11 . The method of claim 8 , wherein the performance counter data comprises a set of normally available counters and one or more normally unavailable counters.
12 . The method of claim 9 , further comprising providing a user with access to the time-series database and the aggregated database simultaneously for real-time and aggregated performance analysis.
13 . The method of claim 11 , wherein the performance counter data comprises instruction counters, cycle counters, page-faults, and context-switches.
14 . The method of claim 12 , wherein the real-time performance analysis comprises creating histograms of the performance counter data.
15 . A non-transitory, computer-readable storage medium storing instructions executable by a processor of a computational device, which when executed cause the computational device to:
execute a performance profiling tool on a computing system;
execute an application on the computing system;
collect performance data about the application from the performance profiling tool;
store the performance data in a database;
monitor an impact of the performance profiling tool on the application; and
adjust an interval at which the performance profiling tool operates in order to keep the impact of the performance profiling tool on the application below a predetermined policy threshold.
16 . The non-transitory, computer-readable storage medium of claim 1515 , which when executed further causes the computational device to:
provide the performance data to one or more system monitors;
receive feedback from the system monitors; and
adjust the interval at which the performance profiling tool operates based on the feedback.
17 . The non-transitory, computer-readable storage medium of claim 1515 , which when executed further causes the computational device to:
provide the performance data to one or more system monitors;
receive feedback from the system monitors; and
add or remove one or more performance measures from being collected based on the feedback.
18 . The non-transitory, computer-readable storage medium of claim 16 , wherein one of the system monitors is a system health monitor.
19 . The non-transitory, computer-readable storage medium of claim 16 , wherein one of the system monitors is a system security monitor.
20 . The non-transitory, computer-readable storage medium of claim 1515 , which when executed further causes the computational device to provide recommendations to a user regarding application optimizations based on the performance data stored in the database.