Time series metric data modeling and prediction
A system that utilizes a plurality of time series of metric data to more accurately detect anomalies and model and predict metric values. Streams of time series metric data are processed to generate a set of independent metrics. In some instances, the present system may automatically analyze thousands of real-time streams. Advanced machine learning and statistical techniques are used to automatically find anomalies and outliers from the independent metrics by learning latent and hidden patterns in the metrics. The trends of each metric may also be analyzed and the trends for each characteristic may be learned. The system can automatically detect latent and hidden patterns of metrics including weekly, daily, holiday and other application specific patterns. Anomaly detection is important to maintaining system health and predicted values are important for customers to monitor and make planning and decisions in a principled and quantitative way.
1. A method for detecting an anomaly in time-series data, comprising:
receiving a plurality of time series of original metric data;
performing a component analysis to generate a plurality of time series of independent metric data;
performing a function to provide coefficients having a varying granularity for the plurality of time series of independent metric data;
predicting a value for the independent metric time series data based on the varying granularity coefficients;
converting the predicted value for the independent metric time series data to an original metric data value; and
determining the actual value of the original metric time series data is an anomaly if the actual value for the metric differs from the predicted value for the metric by more than a threshold.
2. The method of claim 1 , wherein the component analysis includes principal component analysis or independent component analysis.
3. The method of claim 1 , wherein the original metrics include a metric time series generated from monitoring an application and a metric time series generated from monitoring a user behavior.
4. The method of claim 1 , wherein the function is a discrete waveform transformation function, the varying granularity including wavelet coefficients.
5. The method of claim 1 , the method further including generating weighting for each of the coefficients, the predicted value generated based on the weighted coefficients.
6. The method of claim 1 , wherein the predicted value is based on coefficients associated with local data that corresponds to local data associated with the predicted value.
7. The method of claim 6 , wherein the local data includes temporal and date data.
8. The method of claim 1 , further comprising determining an alert based on a detected pattern in a first time series of metric data which is correlated with one or more other time series of metric data.
9. A non-transitory computer readable storage medium having embodied thereon a program, the program being executable by a processor to perform a method for detecting an anomaly in time-series data, the method comprising:
receiving a plurality of time series of original metric data;
performing a component analysis to generate a plurality of time series of independent metric data;
performing a function to provide coefficients having a varying granularity for the plurality of time series of independent metric data;
predicting a value for the independent metric time series data based on the varying granularity coefficients;
converting the predicted value for the independent metric time series data to an original metric data value; and
determining the actual value of the original metric time series data is an anomaly if the actual value for the metric differs from the predicted value for the metric by more than a threshold.
10. The non-transitory computer readable storage medium of claim 9 , wherein the component analysis includes principal component analysis or independent component analysis.
11. The non-transitory computer readable storage medium of claim 9 , wherein the original metrics include a metric time series generated from monitoring an application and a metric time series generated from monitoring a user behavior.
12. The non-transitory computer readable storage medium of claim 9 , wherein the function is a discrete waveform transformation function, the varying granularity including wavelet coefficients.
13. The non-transitory computer readable storage medium of claim 9 , the method further including generating weighting for each of the coefficients, the predicted value generated based on the weighted coefficients.
14. The non-transitory computer readable storage medium of claim 9 , wherein the predicted value is based on coefficients associated with local data that corresponds to local data associated with the predicted value.
15. The non-transitory computer readable storage medium of claim 14 , wherein the local data includes temporal and date data.
16. The non-transitory computer readable storage medium of claim 9 , further comprising determining an alert based on a detected pattern in a first time series of metric data which is correlated with one or more other time series of metric data.
17. A system for detecting an anomaly in time-series data, comprising:
a server including a memory and a processor; and
one or more modules stored in the memory and executed by the processor to receive a plurality of time series of original metric data, perform component analysis to generate a plurality of time series of independent metric data, perform a function to provide coefficients having a varying granularity for the plurality of time series of independent metric data, predict a value for the independent metric time series data based on the varying granularity coefficients, convert the predicted value for the independent metric time series data to an original metric data value, and determine the actual value of the original metric time series data is an anomaly if the actual value for the metric differs from the predicted value for the metric by more than a threshold.
18. The system of claim 17 , wherein the component analysis includes principal component analysis or independent component analysis.
19. The system of claim 17 , wherein the original metrics include a metric time series generated from monitoring an application and a metric time series generated from monitoring a user behavior.
20. The system of claim 17 , wherein the function is a discrete waveform transformation function, the varying granularity including wavelet coefficients.
21. The system of claim 17 , the method further including generating weighting for each of the coefficients, the predicted value generated based on the weighted coefficients.
22. The system of claim 17 , wherein the predicted value is based on coefficients associated with local data that corresponds to local data associated with the predicted value.
23. The system of claim 22 , wherein the local data includes temporal and date data.
24. The system of claim 17 , further comprising determining an alert based on a detected pattern in a first time series of metric data which is correlated with one or more other time series of metric data.