Performance metric collection and automated analysis
View Patent ↗A metric monitoring and analysis system including dynamic sampling agents located in monitored system elements and a service management platform. Each sampling agent includes a data adapter collecting metric data in a common format, a threshold generator for determining dynamic metric threshold ranges, an alarm detector generating an indicator when a metric deviates outside a dynamic threshold range or a static threshold, and a deviation tracker generating an alarm severity scores. The service platform includes an alarm analyzer identifying root causes of system alarm conditions by correlation of grouped metrics or forensic analysis of temporally or statistically correlated secondary forensic data or data items from a service model of the system.
1. A method for performing system metric analysis, comprising the steps of:
collecting metric data representing system operations from a plurality of system sources,
determining a dynamic metric threshold range for each metric over successive time periods,
indicating a metric alarm event by
generating a metric threshold alarm indicator when a corresponding metric value deviates outside the dynamic threshold range,
generating an alarm severity score for each metric alarm event, and
performing a root cause analysis identifying a basic cause of a system alarm condition by at least one of
correlation of grouped metrics with alarm conditions, and
forensic analysis of selected secondary forensic data items recorded upon the occurrence of at least selected alarm conditions.
2. The method for performing system metric analysis of claim 1 , wherein the step of collecting metric data from system sources further includes the steps of:
collecting the metric data from each source in a source format, and
transforming the metric data from the corresponding source format into at least one common format.
3. The method for performing system metric analysis of claim 1 , wherein the step of determining a dynamic metric threshold range for each metric further includes the steps of:
generating an empirical estimate of an actual distribution of values of the metric,
determining a trend in the actual distribution of the values of the metric,
determining an expected range of the values of the metric over the time periods, and
determining a dynamic threshold range for each metric based on the expected range of values of the metric.
4. The method for performing system metric analysis of claim 1 , wherein the step of indicating a metric alarm event further includes the steps of:
generating a metric threshold alarm indicator when a corresponding metric value deviates outside a static threshold range.
5. The method for performing system metric analysis of claim 1 , wherein the step of generating an alarm severity score for each metric alarm further includes the step of:
determining the severity score as a ratio of a time and magnitude of the metric value deviation outside the threshold range to a width of the threshold range.
6. The method for performing system metric analysis of claim 1 , wherein the step of correlation and grouping of metrics with alarm indicators further includes at least one of the steps of:
performing a temporal alarm correlation process by identifying metrics having values resulting in alarm conditions that are correlated in time, and
performing a statistical metric correlation process by correlating pairs of metrics having values resulting in an alarm condition according to the relative ranks of the values of the metrics.
7. The method for performing system metric analysis of claim 1 , wherein the step of analysis of recorded secondary forensic data items and alarm indicators further includes at least one of the steps of:
upon occurrence of each of at least one of the selected alarm indicators,
recording selected key metrics defined as bounding the data items with at least one of temporal and statistical correlation of the metrics and
recording data items selected from a service model of the system representing system operations associated with an alarm condition.
8. The method for performing system metric analysis of claim 1 , wherein the secondary forensic data items comprise non-metric information pertaining to system conditions at occurrence of an alarm condition.
9. The method for performing system metric analysis of claim 1 , wherein:
the steps of
collecting metric data representing system operations from a plurality of system sources,
determining a dynamic metric threshold range for each metric over successive time periods,
indicating a metric alarm event by generating a metric threshold alarm indicator when a corresponding metric value deviates outside the dynamic threshold range, and
generating an alarm severity score for each metric alarm event,
are performed in dynamic sampling agents located at each data source, the step of performing a root cause analysis identifying a basic cause of a system alarm condition is performed
in a central service management platform, and
the method further includes the step of
communicating each metric alarm event and each alarm severity score for each metric alarm event from the dynamic sampling agent to the service management platform.
10. The method for performing system metric analysis of claim 2 , wherein:
the step of collecting metric data from system sources is performed by dynamic sampling agents located at each data source, and
the steps of
collecting the metric data from a source in a source format, and
transforming the metric data from the corresponding source format into at least one common format,
are performed in each dynamic sampling agent by a data adapter.
11. A metric monitoring and analysis system, comprising:
a plurality of dynamic sampling agents, each dynamic sampling agent being located in a system element containing a metric of interest for monitoring and analysis and including
at least one plurality of data collectors for collecting metric data representing system operations from a plurality of system sources,
a threshold generator for determining a dynamic metric threshold range for each metric over successive time periods,
an alarm condition detector indicating a metric alarm event by
generating a metric threshold alarm indicator when a corresponding metric value deviates outside the dynamic threshold range, and
a deviation tracker for generating an alarm severity score for each metric alarm event, and
a single service management platform receiving alarm indicators and severity scores from the dynamic sampling agents and including
an alarm analyzer performing a root cause analysis identifying a basic cause of a system alarm condition by at least one of
correlation of grouped metrics with alarm conditions, and
forensic analysis of selected secondary forensic data items recorded upon the occurrence of at least one or more selected alarm conditions.
12. The metric monitoring and analysis system of claim 11 , wherein each dynamic sampling agent further includes:
at least one data adapter for transforming the metric data from a corresponding source format into at least one common format.
13. The metric monitoring and analysis system of claim 11 , wherein:
for each metric, the threshold generator
generates an empirical estimate of an actual distribution of values of the metric,
determines a trend in the actual distribution of the values of the metric,
determines an expected range of the values of the metric over the time periods, and
determines a dynamic threshold range for each metric based on the expected range of values of the metric.
14. The metric monitoring and analysis system of claim 11 , wherein:
the alarm condition generator generates a metric threshold alarm indicator when a corresponding metric value deviates outside a static threshold range.
15. The metric monitoring and analysis system of claim 11 , wherein:
the deviation tracker generates an alarm severity score for each metric alarm by determining the severity score as a ratio of a time and magnitude of the metric value deviation outside the threshold range to a width of the threshold range.
16. The metric monitoring and analysis system of claim 11 , wherein the alarm analyzer performs a con-elation of grouped metrics by at least one of
a temporal alarm correlation process identifying metrics having values resulting in alarm conditions that are correlated in time, and
a statistical metric correlation process correlating pairs of metrics having values resulting in an alarm condition according to the relative ranks of the values of the metrics.
17. The metric monitoring and analysis system of claim 11 , wherein the alarm analyzer records secondary forensic data items on the occurrence of at least selected alarm indicators by at least one of
recording selected key metrics defined as bounding the data items with at least one of temporal and statistical correlation of the metrics, and
recording data items selected from a service model of the system representing system operations associated with an alarm condition.
18. The metric monitoring and analysis system of claim 12 , wherein the secondary forensic data items comprise non-metric information pertaining to system conditions at occurrence of an alarm condition.