Anomaly detection using circumstance-specific detectors
The technology disclosed relates to learning how to efficiently display anomalies in performance data to an operator. In particular, it relates to assembling performance data for a multiplicity of metrics across a multiplicity of resources on a network and training a classifier that implements at least one circumstance-specific detector used to monitor a time series of performance data or to detect patterns in the time series. The training includes producing a time series of anomaly event candidates including corresponding event information used as input to the detectors, generating feature vectors for the anomaly event candidates, selecting a subset of the candidates as anomalous instance data, and using the feature vectors for the anomalous instance data and implicit and/or explicit feedback from users exposed to a visualization of the monitored time series annotated with visual tags for at least some of the anomalous instances data to train the classifier.
1. A method of efficiently displaying anomalies in performance data to an operator, the method including:
assembling performance data for a multiplicity of metrics across a multiplicity of resources on a network; and
training a machine learning classifier to classify data from at least one circumstance-specific detector used to detect patterns in a time series of the performance data, the training including:
automatically producing a time series of anomaly event candidates including corresponding event information using the circumstance-specific detector;
automatically generating feature vectors using the anomaly event candidates;
automatically selecting a subset of the anomaly event candidates as anomalous instance data;
using the feature vectors for the anomalous instance data and user feedback from users exposed to a visualization of the performance data time series annotated with visual tags for at least some of the anomalous instances data to train the machine learning classifier to distinguish true anomalies from false anomalies; and
efficiently displaying only the true anomalies as distinguished by the machine learning classifier.
2. The method of claim 1 , wherein the circumstance-specific detector includes a trend model that represents trend functions of the time series of the performance data.
3. The method of claim 1 , wherein the circumstance-specific detector includes a periodic model that represents periodic variations in the time series of the performance data.
4. The method of claim 1 , wherein the circumstance-specific detector includes a noise model that represents random or systematic fluctuations in the time series of the performance data.
5. The method of claim 1 , wherein the circumstance-specific detector includes a weighted moving average that estimates a smoothed average of values in the time series of the performance data.
6. The method of claim 1 , wherein the circumstance-specific detector includes an exponentially weighted average that estimates a smoothed average of the time series of the performance data.
7. The method of claim 1 , wherein the circumstance-specific detector includes an autoregressive moving average (ARMA) to predict instances in the time series of the performance data.
8. The method of claim 1 , wherein the circumstance-specific detector includes an autoregressive integrated moving average (ARIMA) to predict instances in the time series of the performance data.
9. The method of claim 1 , wherein the circumstance-specific detector includes a Hidden Markov Model (HMM) that infers stage changes in the time series of the performance data and identifies at least some state changes as anomalies.
10. The method of claim 1 , wherein the circumstance-specific detector treats instances in the times series of the performance data as a Gaussian Process and uses deviations from a mean outcome of the Gaussian Process to identify specific instances as anomalies.
11. The method of claim 1 , wherein the circumstance-specific detector generates a frequency domain decomposition of the time series of the performance data.
12. The method of claim 1 , wherein the circumstance-specific detector is a wavelet analysis that generates a time-frequency domain decomposition of the time series of the performance data.
13. The method of claim 1 , further including:
identifying a set of instances in the time series of the performance data within a predetermined time window as an unordered set of random samples;
estimating at least one of a histogram and an empirical cumulative density function (CDF) using the random samples;
comparing empirical CDFs of at least two or more time-varying instances and determining similarity of underlying probability distribution between the time-varying instances using two-sided Kolmogorov-Smirnov (KS) test; and
determining anomalies in the time series of the performance data by thresholding a KS distance that marks predetermined deviations as change points.
14. The method of claim 1 , wherein the multiplicity of metrics includes at least one of a system level metrics and a service level metrics, including at least one of:
central processing unit (CPU) usage;
disk usage;
memory usage;
process fork rate;
network usage;
system load;
system service level agreements (SLAs);
Hadoop block read latency;
MongoDB read latency; and
Nginx requests per second.
15. The method of claim 1 , further including using the user feedback from users further includes receiving at least one of emoticons, likes, stars, thumbs up, bonuses, ratings, and badges corresponding to a time series of feature vectors.
16. The method of claim 1 , further including using the feature vectors to identify interrelated anomaly event candidates based on at least:
one common time-window within which the anomaly event candidates were detected;
one common metric for which the anomaly event candidates were detected;
one common resource on which the anomaly event candidates occurred;
two or more connected metrics on which two or more respective anomaly event candidates were detected; and
two or more connected resources on which two or more respective anomaly event candidates occurred.
17. The method of claim 1 , wherein using the user feedback from users further includes capturing users' navigation actions during evaluation of a ranked list of anomalies generated using a time series of feature vectors.
18. The method of claim 1 , wherein using the user feedback from users further includes determining users' attention duration to respective anomalies during evaluation of a ranked list of anomalies generated using a time series of feature vectors.
19. A system including one or more processors coupled to memory, the memory loaded with computer instructions to efficiently display anomalies in performance data to an operator, which instructions, when executed on the processors, implement actions comprising:
assembling performance data for a multiplicity of metrics across a multiplicity of resources on a network; and
training a machine learning classifier to classify data from at least one circumstance-specific detector used to detect patterns in a time series of the performance data, the training including:
automatically producing a time series of anomaly event candidates including corresponding event information using the circumstance-specific detector;
automatically generating feature vectors using the anomaly event candidates;
automatically selecting a subset of the anomaly event candidates as anomalous instance data;
using the feature vectors for the anomalous instance data and user feedback from users exposed to a visualization of the performance data time series annotated with visual tags for at least some of the anomalous instances data to train the machine learning classifier to distinguish true anomalies from false anomalies; and
efficiently displaying only the true anomalies as distinguished by the machine learning classifier.
20. A non-transitory computer readable storage medium impressed with computer program instructions to learn how to efficiently display anomalies in performance data to an operator, the instructions, when executed on a processor, implement a method comprising:
assembling performance data for a multiplicity of metrics across a multiplicity of resources on a network; and
training a machine learning classifier to classify data from at least one circumstance-specific detector used to detect patterns in a time series of the performance data, the training including:
automatically producing a time series of anomaly event candidates including corresponding event information using the circumstance-specific detector;
automatically generating feature vectors using the anomaly event candidates;
automatically selecting a subset of the anomaly event candidates as anomalous instance data;
using the feature vectors for the anomalous instance data and user feedback from users exposed to a visualization of the performance data time series annotated with visual tags for at least some of the anomalous instances data to train the machine learning classifier to distinguish true anomalies from false anomalies; and
efficiently displaying only the true anomalies as distinguished by the machine learning classifier.