Time series metric data modeling and prediction
A system that utilizes a plurality of time series of metric data to more accurately detect anomalies and model and predict metric values. Streams of time series metric data are processed to generate a set of independent metrics. In some instances, the present system may automatically analyze thousands of real-time streams. Advanced machine learning and statistical techniques are used to automatically find anomalies and outliers from the independent metrics by learning latent and hidden patterns in the metrics. The trends of each metric may also be analyzed and the trends for each characteristic may be learned. The system can automatically detect latent and hidden patterns of metrics including weekly, daily, holiday and other application specific patterns. Anomaly detection is important to maintaining system health and predicted values are important for customers to monitor and make planning and decisions in a principled and quantitative way.
1. A method for detecting an anomaly in time series data, comprising:
receiving, by a machine, a plurality of time series of original metric data associated with different types of monitoring;
generating, by a component analyzer on the machine, a plurality of time series of independent metric data using the received plurality of time series of original metric data;
generating, by a wavelet engine on the machine, coefficients using the plurality of time series of independent metric data;
determining, by the wavelet engine, weightings for each of the coefficients;
predicting, by a metric prediction module on the machine, a value for the plurality of time series of independent metric data for a future time point using the coefficients and the weightings;
determining, by the metric prediction module, a predicted value for the plurality of time series of original metric data using the predicted value for the plurality of time series of independent metric data; and
detecting, by a pattern analysis module on the machine, an anomaly in the plurality of time series of original metric data by comparing an actual value for the plurality of time series of original metric data received at the future time point with the predicted value for the plurality of time series of original metric data to determine whether the actual value differs from the predicted value by more than a threshold.
2. The method of claim 1 , wherein generating the plurality of time series of independent metric data includes performing principal component analysis or independent component analysis.
3. The method of claim 1 , wherein the plurality of time series of original metric data include a time series of metric data associated with monitored application and a time series of metric data associated with monitored user behavior.
4. The method of claim 1 , wherein generating the coefficients include performing a discrete waveform transformation function.
5. The method of claim 1 , wherein the coefficients include different levels of granularity.
6. The method of claim 1 , further comprising providing, by the pattern analysis module, an alert for the detected anomaly.
7. A system for detecting an anomaly in time-series data, comprising:
a server including a memory and a processor; and
one or more modules stored in the memory and executed by the processor to perform operations including:
generating, by a component analyzer, a plurality of time series of independent metric data by processing a plurality of time series of original metric data associated with different types of monitored processes;
generating, by a wavelet engine, coefficients using the plurality of time series of independent metric data;
determining, by the wavelet engine, weightings for each of the coefficients;
predicting, by a metric prediction module, a predicted value of the independent metric time series data for a future time point using the coefficients and the weightings;
detecting, by a pattern analysis engine, a latent pattern in the plurality of time series of original metric data using the generated plurality of time series of independent metric data based on the predicted value of the independent metric time series data that that are predicted using the coefficients and the weightings;
determining, by the pattern analysis engine, one or more anomalies using the detected pattern; and
providing an alert for the determined one or more anomalies.
8. The system of claim 7 , wherein a number of the plurality of time series of independent metric data is different from a number of the plurality of time series of original metric data associated with different types of monitored processes.
9. The system of claim 7 , wherein the detected pattern is used to detect a condition.
10. The system of claim 9 , wherein the condition includes a performance degradation condition.
11. The system of claim 7 , wherein the detected pattern includes an application specific pattern.
12. The system of claim 7 , wherein detecting the pattern includes detecting a pattern in one metric of a set of correlated metrics in the plurality of time series of original metric data.
13. The system of claim 7 , including:
determining a predicted value of the original metric data for the future time point using the predicted value of the independent metric time series data; and
comparing an actual value of the original metric data received at the future time point with the predicted value of the original metric data.
14. The system of claim 7 , including:
comparing the detected pattern against a stored pattern.
15. A non-transitory computer readable storage medium having embodied thereon a program, the program being executable by a processor to perform a method for detecting an anomaly in time series data, the method comprising:
processing, by component analyzer, a plurality of time series of original metric data associated with different types of monitored processes, wherein the plurality of time series of original metric data is processed by:
generating, by the component analyzer, a plurality of time series of independent metric data by processing a plurality of time series of original metric data associated with different types of monitored processes, and
generating, by a wavelet engine, coefficients using the plurality of time series of independent metric data;
determining, by the wavelet engine, weightings for each of the coefficients;
detecting, by a pattern analysis module, a latent pattern in a set of correlated metrics in the plurality of time series of original metric data using the coefficients and the weightings;
comparing, by a pattern analysis module, the detected pattern against a stored pattern;
detecting a condition based on the comparison; and
providing, by the pattern analysis module, an alert for the detected condition, wherein the alert includes a notification to monitor a specific resource, application, or network associated with the set of correlated metrics.
16. The non-transitory computer readable storage medium of claim 15 , wherein the set of correlated metrics include a central processing unit usage, database processing time, and an application response time.