Smarter performance alerting mechanism combining thresholds and historical seasonality
A storage system management application receives reports describing operation of a storage system from a performance monitoring system, and compares the storage system operating metrics against a set of respective thresholds. If the metrics breach one or more thresholds, the seasonality boundaries for the respective metric are calculated, and an alert is only generated if the storage system operating metric is both outside the performance threshold and outside of the calculated seasonality boundaries. By determining the seasonality boundaries and using the seasonality boundaries to determine when to generate alerts, it is possible to reduce the number of alerts generated. By including the seasonality values in the alerts, the deviation from normal seasonably operating conditions can be determined from the alert, thus enabling a person who receives the alert to determine the magnitude of deviation of the storage system operation from seasonably adjusted normal operation.
1. A method of implementing a performance alerting mechanism combining thresholds and historical seasonality, the method comprising:
receiving, from a host device, an IO command;
sending, responsive to determining that the IO command cannot be serviced by memory of the host device, the IO command to a storage system; and
monitoring performance of the storage system;
wherein monitoring the storage system comprises:
collecting, during a monitoring time interval, storage system performance data of that is a plurality of operational metrics of a storage system;
comparing the collected storage system performance data with a plurality of threshold values to determine whether one or more of the plurality of operational metrics has breached its respective threshold value in the plurality of threshold values;
determining that a first operational metric, of the plurality of operational metrics, has breached its first respective threshold of the plurality of threshold values;
responsive to determining that the first operational metric has breached the first respective threshold:
obtaining historic performance data, of the storage system, of the first operational metric, the historic performance data is over a preceding period of time;
calculating, based on the obtained historic performance data, seasonality boundaries of the first operational metric, in time interval in which the first respective threshold breach occurred; and
determining that the first operational metric is outside the calculated seasonality boundaries; and
generating an alert responsive to determining that the first operational metric is outside the calculated seasonality boundaries.
2. The method of claim 1 , wherein the alert contains a value of the first operational metric, the first respective threshold, and the calculated seasonality boundaries of the first operational metric.
3. The method of claim 1 , wherein at least some of the plurality of thresholds are expressed as bandwidth, response time, or throughput.
4. The method of claim 1 , wherein the historic performance data comprises a timeseries data of the first operational metric, and wherein the step of calculating the seasonality boundaries comprises decomposing the timeseries data to determine a seasonal component, a trend component, and a random component of the timeseries.
5. The method of claim 4 , wherein the step of decomposing the timeseries is implemented using a Seasonal and Trend Decomposition using Loess.
6. The method of claim 4 , wherein the seasonal component reflects periodic variation of the first operational metric.
7. The method of claim 1 , wherein the preceding period of time is a two week or longer period of time.
8. A system, comprising:
a host device and a storage system;
the host device being configured to receive an IO command, determine whether the IO command can be serviced from memory of the host device, and responsive to determining that the IO command cannot be serviced by memory of the host device, send the IO command to the storage system;
the storage system being configured to receive the TO command from the host device and respond to the TO command;
the storage system including a storage system performance monitoring system configured to collect, during a monitoring time interval, storage system performance data that is a plurality of operational metrics of the storage system, and generate reports of storage system operational metrics at end of the monitoring interval; and
the storage system further including a storage system management application configured to receive the reports of storage system operational metrics, and compare the collected storage system performance data with a plurality of threshold values to determine whether one or more of the plurality of operational metrics has breached its respective threshold value in the plurality of threshold values, the storage system management application further comprising control logic configured to:
if a first operational metric, of the plurality of operational metrics, has breached its first respective threshold of the plurality of threshold values:
obtain historic performance data, of the storage systems the first operational metric, the historic performance data is over a preceding period of time;
calculate, based on the obtained historic performance data, seasonality boundaries of the first operational metric, in time interval in which the first respective threshold breach occurred; and
generate an alert only when the first operational metric is outside the calculated seasonality boundaries.
9. The system of claim 8 , wherein the alert contains a value of the first operational metric, the first respective threshold, and the calculated seasonality boundaries of the first operational metric.
10. The system of claim 8 , wherein at least some of the plurality of thresholds are expressed as bandwidth, response time, or throughput.
11. The system of claim 8 , wherein the historic performance data comprises a timeseries data of the first operational metric, and wherein the control logic configured to calculate the seasonality boundaries comprises control logic configured to decompose the timeseries data to determine a seasonal component, a trend component, and a random component of the timeseries.
12. The system of claim 11 , wherein the logic configured to decompose the timeseries is configured to perform a Seasonal and Trend Decomposition using Loess.
13. The system of claim 11 , wherein the seasonal component reflects periodic variation of the first operational metric.
14. The system of claim 8 , wherein the period of time preceding the monitoring interval is a two week or longer period of time.
15. A method of implementing a performance alerting mechanism combining thresholds and historical seasonality, comprising:
setting a plurality of QOS metrics on a storage system by a storage system management application, the QOS metrics specifying required storage system operation parameters;
receiving, from a host device, an IO command;
sending, responsive to determining that the TO command cannot be serviced by memory of the host device, the TO command to the storage system; and
monitoring performance of the storage system;
wherein monitoring the storage system comprises:
collecting, during a monitoring time interval, storage system performance data that is a plurality of operational metrics of the storage system by a storage system performance monitoring system, the plurality of operational metrics including operational characteristics of physical aspects of the storage system and operational metrics of logical aspects of the storage system;
providing reports describing operation of the storage system by the storage system performance monitoring system to the storage system management application;
comparing, by the storage system management application, the collected storage system performance data with a plurality of threshold values to determine whether one or more of the plurality of operational metrics has breached its respective threshold value in the plurality of threshold values;
determining that a first operational metric, of the plurality of operational metrics, has breached its first respective threshold of the plurality of threshold values;
responsive to determining that the first operational metric has breached the first respective threshold:
obtaining historic performance data, of the storage system, of the first operational metric, the historic performance data is over a preceding period of time;
calculating, based on the obtained historic performance data, seasonality boundaries of the first operational metric, in time interval in which the first respective threshold breach occurred; and
determining that the first operational metric is outside the calculated seasonality boundaries; and
generating an alert by the storage system management application responsive to determining that the first operational metric is outside the calculated seasonality boundaries.
16. The method of claim 15 , wherein the alert contains a value of the first operational metric, the first respective threshold, and the calculated seasonality boundaries of the first operational metric.
17. The method of claim 15 , wherein at least some of the plurality of thresholds are expressed as bandwidth, response time, or throughput.
18. The method of claim 15 , wherein the historic performance data comprises a timeseries data of the first operational metric, and wherein the control logic configured to calculate the seasonality boundaries comprises control logic configured to decompose the timeseries data to determine a seasonal component, a trend component, and a random component of the timeseries.
19. The method of claim 18 :
wherein the logic configured to decompose the timeseries is configured to perform a Seasonal and Trend Decomposition using Loess; and
wherein the seasonal component reflects periodic variation of the first operational metric.
20. The method of claim 15 , wherein the period of time preceding the monitoring interval is a two week or longer period of time.