Anomaly detection for non-stationary data
A method of detecting anomalies in a time series is disclosed. A training time series corresponding to a process is extracted from an initial time series corresponding to the process, the training time series including a subset of the initial time series. Outlier data points in the training time series are modified based on predetermined acceptability criteria. A plurality of prediction methods are trained using the training time series. An actual data point corresponding to the initial time series is received. The plurality of prediction methods are used to determine a set of predicted data points corresponding to the actual data point. It is determined whether the actual data point is anomalous based on a calculation of whether each of the set of predicted data points is statistically different from the actual data point.
1. A method comprising:
incorporating one or more anomaly detection applications into a computing system, the one or more anomaly detection applications configuring one or more computer processors of the computing system to perform operations for generating a user interface for representing a health of a process executing within the computing system, the operations comprising:
receiving an initial time series of data points;
receiving a training time series of data points representing a recent subset of data points in the initial time series of data points;
modifying an outlier data point in the training time series of data points to bring the outlier data point into a range of other data points in the recent subset of data points, the modifying of the outlier data point comprising capping the outlier data point;
training at least one prediction method using at least the training time series of data points;
measuring a first actual data point, the first actual data point being based on data at a later time than the last data point in the training time series;
using the at least one prediction method to determine a predicted value corresponding to the first actual data point;
determining whether the first actual data point is anomalous based on a calculation of whether the first actual data point is statistically different from the predicted data point; and
performing the generating of the user interface, the generating comprising providing a visual representation comprising the first actual data point, the predicted data point, a visual indication of a first determination of corresponding to whether the first actual data point is anomalous, a visual indication of a second determination corresponding to whether the second actual data point is anomalous, the visual indication of the second determination representing a relative strength of the second determination, the relative strength of the second determination being represented by a size of the visual indication of the second determination relative to a size of the visual indication of the first determination.
2. The method of claim 1 , wherein the visual representation includes a color associated with the determination of whether the actual data point is anomalous.
3. The method of claim 2 , wherein the visual representation includes time on an x-axis of a chart.
4. The method of claim 3 , wherein the visual representation includes a value of a metric on a y-axis of the chart.
5. The method of claim 1 , wherein the calculation of whether the actual data point is statistically different from the predicted data point includes a determination of a standard deviation.
6. The method of claim 1 , wherein the initial time series corresponds to a category of data generated by one or more processes executing on a system.
7. A system comprising:
one or more computer processors; and
one or more computer memories storing a set of instructions that cause the one or more computer processors to perform operations comprising:
incorporating one or more anomaly detection applications into a computing system, the one or more anomaly detection applications configuring one or more computer processors of the computing system to perform operations for generating a user interface for representing a health of a process executing within the computing system, the operations comprising:
receiving an initial time series of data points;
receiving a training time series of data points representing a recent subset of data points in the initial time series of data points;
modifying an outlier data point in the training time series of data points to bring the outlier data point into a range of other data points in the recent subset of data points, the modifying of the outlier data point comprising capping the outlier data point;
training at least one prediction method using at least the training time series of data points;
measuring a first actual data point, the first actual data point being based on data at a later time than the last data point in the training time series;
using the at least one prediction method to determine a predicted value corresponding to the first actual data point;
determining whether the first actual data point is anomalous based on a calculation of whether the first actual data point is statistically different from the predicted data point; and
performing the generating of the user interface, the generating comprising providing a visual representation comprising the first actual data point, the predicted data point, a visual indication of a first determination corresponding to whether the first actual data point is anomalous, a visual indication of a second determination corresponding to whether the second actual data point is anomalous, the visual indication of the second determination representing a relative strength of the second determination, the relative strength of the second determination being represented by a size of the visual indication of the second determination relative to a size of the visual indication of the first determination.
8. The system of claim 7 , wherein the visual representation includes a color associated with the determination of whether the actual data point is anomalous.
9. The system of claim 8 , wherein the visual representation includes time on an x-axis of a chart.
10. The system of claim 9 , wherein the visual representation includes a value of a metric on a y-axis of the chart.
11. The system of claim 7 , wherein the calculation of whether the actual data point is statistically different from the predicted data point includes a determination of a standard deviation.
12. The system of claim 7 , wherein the initial time series corresponds to a category of data generated by one or more processes executing on a system.
13. A non-transitory machine-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
incorporating one or more anomaly detection applications into a computing system, the one or more anomaly detection applications configuring one or more computer processors of the computing system to perform operations for generating a user interface for representing a health of a process executing within the computing system, the operations comprising:
receiving an initial time series of data points;
receiving a training time series of data points representing a recent subset of data points in the initial time series of data points;
modifying an outlier data point in the training time series of data points to bring the outlier data point into a range of other data points in the recent subset of data points, the modifying of the outlier data point comprising capping the outlier data point;
training at least one prediction method using at least the training time series of data points;
measuring first actual data point, the first actual data point being based on data at a later time than the last data point in the training time series;
using the at least one prediction method to determine a predicted value corresponding to the first actual data point;
determining whether the first actual data point is anomalous based on a calculation of whether the first actual data point is statistically different from the predicted data point; and
performing the generating of the user interface, the generating comprising providing a visual representation comprising the first actual data point, the predicted data point, a visual indication of a first determination corresponding to whether the first actual data point is anomalous, a visual indication of a second determination corresponding to whether the second actual data point is anomalous, the visual indication of the second determination representing a relative strength of the second determination, the relative strength of the second determination being represented by a size of the visual indication of the second determination relative to a size of the visual indication of the first determination.
14. The non-transitory machine-readable storage medium of claim 13 , wherein the visual representation includes a color associated with the determination of whether the actual data point is anomalous.
15. The non-transitory machine-readable storage medium of claim 14 , wherein the visual representation includes time on an x-axis of a chart.
16. The non-transitory machine-readable storage medium of claim 15 , wherein the visual representation includes a value of a metric on a y-axis of the chart.
17. The non-transitory machine-readable storage medium of claim 13 , wherein the calculation of whether the actual data point is statistically different from the predicted data point includes a determination of a standard deviation.