Systems and methods for robust anomaly detection

1. A system, comprising:

a distributed cache configured to store state information for a plurality of configuration items (CIs);

a plurality of management, instrumentation, and discovery (MID) servers forming a cluster, each of the plurality of MID servers comprising:

one or more processors, configured to execute machine-readable instructions;

a tangible, non-transitory, machine-readable medium, comprising the machine-readable instructions that, when executed by the one or more processors, cause a corresponding MID server to:

receive, from the distributed cache, a subset of the state information associated with assigned CIs; and

perform a statistical analysis on the subset of the state information; and

an anomaly detector, configured to:

identify statistical outliers of the statistical analysis; and

identify an anomaly of the statistical outliers by tracking a history of the statistical outliers; wherein the anomaly is determined based upon a magnitude of deviation between the subset of the state information and a statistical model;

a change detector, configured to:

identify the statistical outliers using a first filter applied to the subset of the state information, the first filter comprising a Kalman filter that produces estimates of unknown variables using Bayesian inference and joint probability distribution estimation over the unknown variables for a timeframe;

determine when a data transition associated with the statistical outliers indicates noise;

determine when the data transition associated with the statistical outliers indicates a level change, by

upon identifying the statistical outliers, feed data to a second filter that represents data indicative of no statistical outlier occurring;

upon stabilization of a change in the first filter, use an output of the second filter as a reference to determine is the change is statically significant;

when the change is statistically significant, classify the data transition associated with the statistical outliers as a level shift, and

otherwise, when the change is not statistically significant, classify the data transition associated with the statistical outliers as noise; and

when the data transition associated with the statistical outliers indicates the level change, present a notification of the level change.

2. The system of claim 1 , wherein each of the MID servers comprises:

a time-series analyzer, configured to:

classify the subset of the state information into at least one of a plurality of classifications based at least in part upon historical time-series data; and

construct a statistical model representative of a stream of current time-series data based upon the at least one of the plurality of classifications;

an anomaly detector, configured to:

monitor the stream of current time-series data;

perform a statistical analysis on the stream of the current time-series data based at least in part upon the statistical model constructed by the time-series analyzer;

identify statistical outliers of the stream of the current time-series data based at least on the statistical analysis;

determine an anomalous score for the statistical outliers based at least on a history of the statistical outliers; wherein the anomalous score indicates a magnitude of deviation between the current time-series data and the statistical model; and

present the anomalous score to a client communicatively coupled to the system, for subsequent reporting, client action, or both.

3. A system comprising,

a distributed cache configured to store state information for a plurality of configuration items (CIs);

a plurality of management, instrumentation, and discovery (MID) servers forming a cluster, each of the plurality of MID servers comprising:

one or more processors, configured to execute machine-readable instructions;

a tangible, non-transitory, machine-readable medium, comprising the machine-readable instructions that, when executed by the one or more processors, cause a corresponding MID server to:

receive, from the distributed cache, a subset of the state information associated with assigned CIs; and

perform a statistical analysis on the subset of the state information; and

a time-series analyzer, configured to:

classify the subset of the state information into at least one of a plurality of classifications based at least in part upon historical time-series data; and

construct a statistical model representative of a stream of current time-series data based upon the at least one of the plurality of classifications;

an anomaly detector, configured to:

monitor the stream of current time-series data;

perform a statistical analysis on the stream of current time-series data based at least in part upon the statistical model constructed by the time-series analyzer;

identify statistical outliers of the stream of current time-series data based at least on the statistical analysis;

determine an anomalous score for the statistical outliers based at least on a history of the statistical outliers; wherein the anomalous score indicates a magnitude of deviation between the current time-series data and the statistical model; and

present the anomalous score to a client communicatively coupled to the system, for subsequent reporting, client action, or both;

wherein the plurality of classifications comprise:

a quasi-normal classification, indicating that the subset of the state information appears to adhere to a stationary process;

a seasonal classification, indicating that the subset of the state information appears to adhere to a process having periodic non-normal distribution, a general trend over time, or both;

a near-constant classification, indicating that the subset of the state information appears to adhere to a constant value;

a trendy classification, indicating that the subset of the state information comprises noisy data with a trend;

an accumulator classification, indicating that the subset of the state information comprises an accumulation of low-noise data;

a categorical classification, indicating that the subset of the state information appears to adhere to a process having a set of discrete values;

a switching classification, indicating that the subset of the state information appears to adhere to a process of jumping from at least one first state to at least one second state via one or more state transitions; and

an arrival classification, indicating that the subset of the state information appears to adhere to a process dependent upon a timing of a particular event.

4. The system of claim 3 , wherein the one or more processors determine whether the subset of the state information adheres to a weekly model or a daily model when the subset of the state information is seasonal.

5. The system of claim 4 , wherein the one or more processors determine whether the subset of the state information adheres to the weekly model or the daily model by comparing a log-likelihood for a fit of the weekly model and the daily model to a fit threshold.

6. The system of claim 5 , wherein the fit threshold is identified based upon Wilks' Theorem.

7. A tangible, non-transitory, machine-readable medium, comprising machine-readable instructions that, when executed by one or more processors, cause the one or more processors to:

receive a stream of current time-series data;

at periodic intervals, classify the stream of current time-series data into at least one of a plurality of classifications based at least in part upon historical time-series data;

identify a statistical model representative of the stream of current time-series data based upon the classification of the stream;

when the classification comprises a seasonal classification, identify the statistical model representative of the stream of current time-series data from a selection between a weekly statistical model and a daily statistical model; and

perform a statistical analysis on the stream of the current time-series data based at least in part upon the statistical model, wherein the statistical model is constructed by a time-series analyzer;

identify a statistical outlier of the stream of the current time-series data based at least on the statistical analysis; and

classify the statistical outlier as either noise or a level shift, by:

feeding unobserved data that that suggests that no statistical outlier has occurred into a filter;

determining, via filtering of the unobserved data by the filter, whether a data transition associated with the statistical outlier is statistically significant;

classifying the statistical outlier as a level shift when the data transition is statistically significant; and

classifying the statistical outlier as noise when the data transition is not statistically significant; and

provide a graphical indication of results of the statistical analysis at a communicatively coupled instance.

8. The machine-readable medium of claim 7 , comprising instructions that, when executed by the one or more processors, cause the one or more processors to:

identify statistical outliers of the stream of the current time-series data based at least on the statistical analysis;

determine an anomalous score for the statistical outliers by tracking a history of the statistical outliers; wherein the anomalous score indicates a magnitude of deviation between the stream of current time-series data and the statistical model; and

store the anomalous score, for subsequent reporting, client action, or both.

9. The machine-readable medium of claim 7 , comprising instructions that, when executed by the one or more processors, cause the one or more processors to:

receive only a subset of the stream of the current time-series data associated with assigned configuration items (CIs).

10. A management, instrumentation, and discovery (MID) server, comprising:

one or more processors, configured to execute machine-readable instructions;

a tangible, non-transitory, machine-readable medium, comprising the machine-readable instructions that, when executed by the one or more, cause the MID server to:

receive a subset of a stream of current time-series data that is associated with assigned configuration items (CIs) of the MID server;

classify the subset of the stream of current time-series data as seasonal data;

select a statistical model representative of the subset of the stream of current time-series data from a selection between a weekly statistical model and a daily statistical model;

perform a statistical analysis on the stream of the current time-series data based at least in part upon the statistical model constructed by a time-series analyzer;

identify statistical outliers of the stream of the current time-series data based at least on the statistical analysis;

classify the statistical outliers as either transient or a level shift using a first Kalman filter; and

when the statistical outliers are classified as transient, identify an anomaly from the statistical outliers by tracking a history of the statistical outliers using a second Kalman filter; wherein the anomaly is identified based upon a magnitude of deviation between the current time-series data and the statistical model.

11. The MID server of claim 10 , wherein the machine-readable medium comprises machine-readable instructions that, when executed by the one or more processors:

cause the MID server to act as a redundant MID server for an offline MID server of a cluster comprising the MID server and the offline MID server.

12. The MID server of claim 10 , comprising a System Center—Operations Manager (SCOM), a Representational State Transfer (REST) application-programming-interface (API), or both that is used to receive the subset of a stream of current time-series data from a distributed cache.