Anomaly detection using circumstance-specific detectors
The technology disclosed relates to learning how to efficiently display anomalies in performance data to an operator. In particular, it relates to assembling performance data for a multiplicity of metrics across a multiplicity of resources on a network and training a classifier that implements at least one circumstance-specific detector used to monitor a time series of performance data or to detect patterns in the time series. The training includes producing a time series of anomaly event candidates including corresponding event information used as input to the detectors, generating feature vectors for the anomaly event candidates, selecting a subset of the candidates as anomalous instance data, and using the feature vectors for the anomalous instance data and implicit and/or explicit feedback from users exposed to a visualization of the monitored time series annotated with visual tags for at least some of the anomalous instances data to train the classifier.
1. A computer implemented method of selecting by one or more hardware processors executing computer instructions, a subset of anomalies from a performance data set for display, the method including the one or more hardware processors:
assembling a performance data set from performance data for a multiplicity of metrics across a multiplicity of resources on a network; and
using a trained machine learning classifier to automatically classify data from at least one circumstance-specific detector used to monitor a monitored time series of performance data or to detect patterns in the time series of the performance data to select from received performance data a subset consisting of anomalous ones for display, wherein the trained machine learning classifier has been trained by one or more hardware processors:
automatically producing a time series of anomaly event candidates including corresponding event information using the circumstance-specific detector;
automatically generating feature vectors using anomaly event candidates of the time series of anomaly event candidates;
automatically selecting a subset of the anomaly event candidates as anomalous instance data; and
automatically using the feature vectors for the anomalous instance data and user feedback from users exposed to a visualization of the monitored time series of performance data annotated with visual tags for at least some of the anomalous instances data to train the trained machine learning classifier; and
the one or more hardware processors displaying only the anomalies as distinguished by the trained machine learning classifier.
2. The method of claim 1 , wherein the circumstance-specific detector includes a trend model that represents trend functions of the monitored time series.
3. The method of claim 1 , wherein the circumstance-specific detector includes a periodic model that represents periodic variations in the monitored time series.
4. The method of claim 1 , wherein the circumstance-specific detector includes a noise model that represents random or systematic fluctuations in the monitored time series.
5. The method of claim 1 , wherein the circumstance-specific detector includes a weighted moving average that estimates a smoothed average of values in the monitored time series.
6. The method of claim 1 , wherein the circumstance-specific detector includes an exponentially weighted average that estimates a smoothed average of the monitored time series.
7. The method of claim 1 , wherein the circumstance-specific detector includes an autoregressive moving average (ARMA) to predict instances in the monitored time series.
8. The method of claim 1 , wherein the circumstance-specific detector includes an autoregressive integrated moving average (ARIMA) to predict instances in the monitored time series.
9. The method of claim 1 , wherein the circumstance-specific detector includes a Hidden Markov Model (HMM) that infers stage changes in the monitored time series and identifies at least some state changes as anomalies.
10. The method of claim 1 , wherein the circumstance-specific detector treats instances in the monitored times series as a Gaussian Process and uses deviations from a mean outcome of the Gaussian Process to identify specific instances as anomalies.
11. The method of claim 1 , wherein the circumstance-specific detector generates a frequency domain decomposition of the monitored time series.
12. The method of claim 1 , wherein the circumstance-specific detector is a wavelet analysis that generates a time-frequency domain decomposition of the monitored time series.
13. The method of claim 1 , further including:
identifying a set of instances in the monitored time series within a predetermined time window as an unordered set of random samples;
estimating at least one of a histogram and an empirical cumulative density function (CDF) using random samples from the unordered set of random samples;
comparing empirical CDFs of at least two or more time-varying instances and determining similarity of underlying probability distribution between the time-varying instances using two-sided Kolmogorov-Smirnov (KS) test; and
determining anomalies in the monitored time series by thresholding a KS distance that marks predetermined deviations as change points.
14. The method of claim 1 , wherein the multiplicity of metrics includes at least one of a system level metrics and a service level metrics, including at least one of:
central processing unit (CPU) usage;
disk usage;
memory usage;
process fork rate;
network usage;
system load;
system service level agreements (SLAs);
Hadoop block read latency;
MongoDB read latency; and
Nginx requests per second.
15. The method of claim 1 , further including using explicit feedback from users including receiving at least one of emoticons, likes, stars, thumbs up, bonuses, ratings, and badges corresponding to a time series of feature vectors.
16. The method of claim 1 , further including using the feature vectors to identify interrelated anomaly event candidates based on at least:
one common time-window within which the anomaly event candidates were detected;
one common metric for which the anomaly event candidates were detected;
one common resource on which the anomaly event candidates occurred;
two or more connected metrics on which two or more respective anomaly event candidates were detected; and
two or more connected resources on which two or more respective anomaly event candidates occurred.
17. The method of claim 1 , wherein using the user feedback from users further includes capturing users navigation actions during evaluation of a ranked list of anomalies generated using a time series of feature vectors.
18. The method of claim 1 , wherein using the user feedback from users further includes determining users' attention duration to respective anomalies during evaluation of a ranked list of anomalies generated using a time series of feature vectors.
19. A system including one or more processors coupled to memory loaded with computer instructions to select a subset of anomalies from a performance data set for display, which instructions, when executed on the processors, implement actions comprising:
assembling a performance data set from performance data for a multiplicity of metrics across a multiplicity of resources on a network; and
using a trained machine learning classifier to automatically classify data from at least one circumstance-specific detector used to monitor a time series of performance data or to detect patterns in the time series of the performance data to select from received performance data a subset consisting of anomalous ones for display, wherein the trained machine learning classifier has been trained by one or more hardware processors:
automatically producing a time series of anomaly event candidates including corresponding event information using the circumstance-specific detector;
automatically generating feature vectors using anomaly event candidates of the time series of anomaly event candidates;
automatically selecting a subset of the anomaly event candidates as anomalous instance data; and
using the feature vectors for the anomalous instance data and user feedback from users exposed to a visualization of the monitored time series annotated with visual tags for at least some of the anomalous instances data to train the trained machine learning classifier; and
the one or more hardware processors displaying on a display only anomalies as distinguished by the trained machine learning classifier.
20. A non-transitory computer readable storage medium impressed with computer program instructions to select a subset of anomalies from a performance data set for display, which instructions, when executed on one or more hardware processors, implement a method comprising:
assembling a performance data set from performance data for a multiplicity of metrics across a multiplicity of resources on a network; and
using a trained machine learning classifier to automatically classify data from at least one circumstance-specific detector used to monitor a time series of performance data or to detect patterns in the time series of the performance data to select from received performance data a subset consisting of anomalous ones for display, wherein the trained machine learning classifier has been trained by one or more hardware processors:
automatically producing a time series of anomaly event candidates including corresponding event information using the circumstance-specific detector;
automatically generating feature vectors using anomaly event candidates of the time series of anomaly event candidates;
automatically selecting a subset of the anomaly event candidates as anomalous instance data; and
using the feature vectors for the anomalous instance data and user feedback from users exposed to a visualization of the monitored time series annotated with visual tags for at least some of the anomalous instances data to train the trained machine learning classifier; and
the one or more hardware processors displaying on a display only anomalies as distinguished by the trained machine learning classifier.