Methods and systems for generation and optimization of metric threshold for anomaly detection
The present disclosure is related to the field of threshold generation where complex temporal and spatial behavior of metrics are modelled to define normal operating baseline of every component in a system environment. Embodiments of the present disclosure provide methods and systems for generation and optimization of metric threshold for anomaly detection. In the present disclosure, temporal properties such as variation, behavioral patterns of a plurality of metrics are derived. The derived temporal properties are used to generate data-driven and domain-aware static or dynamic thresholds. Additionally, the present disclosure factors spatial and temporal properties collectively to mine a role of influencing metrics and define composite thresholds. In order to cater to dynamic behavior of the system environment and changes in business and technological aspects, the system and method of the present disclosure self-learns, self-tunes, and adapts itself based on user feedback which helps in capturing tacit knowledge of domain experts.
1 . A processor implemented method for recommending optimal thresholds for anomaly detection in an information technology (IT) system environment, the method comprising:
receiving, via one or more Input/Output (I/O) interfaces, a plurality of data corresponding to one or more entities in the IT system environment and a plurality of metrics related to the one or more entities, wherein the IT system environment represents represent a setup of a domain agnostic infrastructure of an organization, wherein the one or more entities represent components of the IT system environment, wherein plurality of metrics include a CPU utilization, a memory utilization, a disk utilization, and a process with a start time, an end time and a runtime in a batch system;
detecting, via the one or more hardware processors, a steady state of each of the plurality of metrics with a change-point detection referring to a persistent change in behavior of the plurality of metrics in accordance with internal infrastructural changes, and the change-point detection is performed, prior to analyzing data for recommending the optimal threshold recommendation, to derive recent steady state of the metrics under consideration, and an ensemble of change-point detection algorithms is used to detect changes in a mean, a variation, patterns, and a trend of the metrics;
deriving, via the one or more hardware processors, (i) one or more temporal properties, and (ii) one or more spatio-temporal properties of each of the plurality of metrics in accordance with the detected steady state;
generating, via the one or more hardware processors, (a) a static threshold, (b) a dynamic threshold using temporal analysis, and (c) a composite threshold based on the derived (i) the one or more temporal properties, and (ii) the one or more spatio-temporal properties of each of the plurality of metrics using one or more models, wherein the static threshold is generated based on a mean μ(g), a standard variation σ(g), and an aggressiveness factor α of each of the plurality of metrics related to the one or more entities in the system environment, wherein a value of the aggressiveness factor α is derived based on closeness of data distribution to a normal behavior, wherein when the metric under consideration observes variation, then the static threshold (ST) is defined as:
ST
(
upper
)
=
μ
(
g
)
+
ασ
(
g
)
ST
(
lower
)
=
μ
(
g
)
-
ασ
(
g
)
,
wherein when the static threshold values are derived using behavioral properties of metrics, a plurality of domain constraints including upper limit of thresholds (UL) and lower limit of thresholds (LL) are factored to recommend an optimal threshold value computed as:
ST
(
upper
)
=
max
(
ST
(
upper
)
,
LL
)
ST
(
upper
)
=
min
(
ST
(
lower
)
,
UL
)
,
wherein the dynamic threshold represents temporal threshold recommendation that is generated when a metric of the plurality of metrics under consideration observes significant variations across varied temporal dimensions and is derived and recommended when the metric under consideration observes significant variations across varied temporal dimensions and follows a specific pattern;
detecting, via the one or more hardware processors, one or more anomalies in the received plurality of data when a value of the generated at least one of (a) the static threshold, (b) the dynamic threshold, and (c) the composite threshold is exceeded to a predefined value;
adaptively updating, via the one or more hardware processors, the one or more models to the at least one of (i) the one or more temporal properties, and (ii) the one or more spatio-temporal properties of each of the plurality of metrics related to the one or more entities in the IT system environment based on at least one of (a) a self-learning mechanism, and (b) a self-tuning mechanism,
wherein in the self-learning mechanism, one or more parameters associated with the plurality of metrics, including a temporal pattern, a level of aggressiveness, an intra-bucket variation (ITA), an inter-bucket variation (ITR), and a score generated using ITA and ITR, in accordance with one or more inputs obtained as a feedback from one or more users are learnt and replicates the learning on each of the plurality of metrics related to the one or more entities in the IT system environment,
wherein in the self-tuning mechanism, group of similar metrics are identified, and corresponding set of representative metrics for a cluster of the set of similar metrics are identified, followed by statistics derived from a previous run of temporal analysis to combine with a statistics of a real time delta data; and
applying the adaptively updated models across domains to suppress false alerts and capture true positives.
2 . The processor implemented method of claim 1 , further comprising:
continuously monitoring, a mismatch between the generated at least one of (a) the static threshold, (b) the dynamic threshold, and (c) the composite threshold with a pre-determined threshold; and
recommending, a corrected threshold when a mismatch between the generated at least one of (a) the static threshold, (b) the dynamic threshold, and (c) the composite threshold and the pre-determined threshold is detected.
3 . The processor implemented method of claim 1 , wherein generating the dynamic threshold comprises:
resampling and categorizing the plurality of data into a plurality of buckets for a plurality of dimensions of the one or more temporal properties of each of the plurality of metrics related to the one or more entities in the system environment;
computing a representative value for each bucket from the plurality of buckets, wherein the representative value is computed based on a mean, and a standard deviation for each bucket from the plurality of buckets for each dimension from the plurality of dimensions;
computing an intra-bucket variation (ITA) for the plurality of data and inter-bucket variation (ITR) for the representative value for each bucket from the plurality of buckets, wherein the intra-bucket variation (ITA) captures average spread of a plurality of data points within each bucket from the plurality of buckets and the inter-bucket variation (ITR) captures variation of the plurality of data points across the plurality of dimensions;
generating a score for the one or more temporal properties of each of the plurality of metrics related to the one or more entities in the system environment based on the ITR and ITA;
identifying an optimal temporal property from the one or more temporal properties of each of the plurality of metrics related to the one or more entities in the system environment based on the generated score; and
generating the dynamic threshold based on the mean, the standard variation and an aggressiveness factor of each of the plurality of metrics related to the one or more entities in the system environment in accordance with the identified temporal property from the one or more temporal properties as
DT
i
(
upper
)
=
μ
(
b
)
i
+
α
σ
(
b
)
i
∀
i
∈
1
,
2
,
…
N
(
b
)
DT
i
(
lower
)
=
μ
(
b
)
i
-
α
σ
(
b
)
i
∀
i
∈
1
,
2
,
…
N
(
b
)
,
wherein when the dynamic threshold values are derived using behavioral properties of metrics, a plurality of domain constraints including an upper limit UL and a lower limit LL are factored to recommend an optimal threshold value for each dimension value computed as:
D
T
i
=
max
(
D
T
i
,
LL
)
∀
i
∈
1
,
2
,
…
N
(
b
)
D
T
i
=
min
(
D
T
i
,
UL
)
∀
i
∈
1
,
2
,
…
N
(
b
)
.
4 . The processor implemented method of claim 1 , wherein generating the composite threshold comprises:
receiving, a first set of events corresponding to an entity identified from the one or more entities in the system environment with no metric;
mining, using one or more data sources, an influencing relationship between a node of the identified entity and a plurality of influencing metrics corresponding to the node of the identified entity to define a set of spatial dependencies;
applying a temporal threshold analysis to each influencing metric from the plurality of influencing metrics to generate at least one of a (i) a respective static threshold, (ii) a respective dynamic threshold, and (iii) a second set of events;
correlating the second set of events with the first set of events occurring in the node of the identified entity to access a plurality of temporal correlations; and
generating the composite threshold based on the plurality of temporal correlations, wherein the plurality of temporal correlations are indicative of behavior of the plurality of influencing metrics.
5 . The processor implemented method of claim 1 , wherein the self-learning mechanism comprises:
generating, a plurality of clusters of the plurality of metrics related to the one or more entities based on a plurality of metric properties;
identifying a set of representative metrics for each cluster from the plurality of clusters of the plurality of metrics based on a similarity in the plurality of metric properties;
generating, at least one of (i) a plurality of optimal static thresholds, (ii) a plurality of optimal dynamic thresholds, and (iii) a plurality of optimal composite thresholds across varying degree of aggressiveness for each representative metric from the set of representative metrics for each cluster from the plurality of clusters of the plurality of metrics;
selecting, an optimal threshold from the at least one of (i) the plurality of optimal static thresholds, (ii) the plurality of optimal dynamic thresholds, and (iii) the plurality of optimal composite thresholds by applying one or more inputs obtained as feedback from one or more users includes user's preferred dimensions, user-defined threshold, and user's preferred aggressiveness factor, on each of the set of representative metrics for each cluster from the plurality of clusters of the plurality of metrics and replicate learning on each of the plurality of metrics related to the one or more entities in the system environment;
determining, a change in one or more parameters associated with the plurality of metrics related to the one or more entities in the system environment based on the one or more inputs obtained as the feedback from the one or more users on each cluster from the plurality of clusters of the plurality of metrics; and
self-learning, the one or more parameters associated with the plurality of metrics in accordance with the one or more inputs obtained as the feedback from the one or more users.
6 . The processor implemented method of claim 1 , wherein the self-tuning mechanism comprises:
obtaining, a plurality of real time delta data corresponding to the one or more entities and a set of metrics related to the one or more entities in the system environment for the plurality of real time delta data;
performing temporal analysis on the set of metrics related to the one or more entities in the system environment for the plurality of real-time delta data, wherein the temporal analysis comprising:
(a) generating, a plurality of clusters of the set of metrics related to the one or more entities in IT system environment for the plurality of real time delta data based on the plurality of metric properties;
(b) identifying a representative metric for each cluster from the plurality of clusters of the set of metrics based on a similarity in the plurality of metric properties; and
(c) generating, at least one of (i) a plurality of optimal static thresholds, (ii) a plurality of optimal dynamic thresholds, and (iii) a plurality of optimal composite thresholds across varying degree of aggressiveness for the representative metric for each cluster from the plurality of clusters of the set of metrics;
determining, a change in the one or more parameters associated with the set of metrics related to the one or more entities in the system environment for the plurality of real time delta data; and
self-tuning, by auto-optimizing the one or more parameters associated with the set of metrics based on (a) the self-learning of the one or more parameters associated with the plurality of metrics and (b) the temporal analysis performed on the set of metrics related to the one or more entities in the system environment for the plurality of real-time delta data.
7 . A system for recommending optimal thresholds for anomaly detection in an information technology (IT) system environment comprising:
a memory storing instructions;
one or more Input/Output (I/O) interfaces; and
one or more hardware processors coupled to the memory via the one or more I/O interfaces, wherein the one or more hardware processors are configured by the instructions to:
receive a plurality of data corresponding to one or more entities in the IT system environment and a plurality of metrics related to the one or more entities, wherein the IT system environment represents represent a setup of a domain agnostic infrastructure of an organization, wherein the one or more entities represent components of the IT system environment, wherein plurality of metrics includes a CPU utilization, a memory utilization, a disk utilization, and a process with a start time, an end time and a runtime in a batch system;
detect a steady state of each of the plurality of metrics with a change-point detection referring to a persistent change in behavior of the plurality of metrics in accordance with internal infrastructural changes, and the change-point detection is performed, prior to analyzing data for recommending the optimal threshold recommendation, to derive recent steady state of the metrics under consideration, and an ensemble of change-point detection algorithms is used to detect changes in a mean, a variation, patterns, and a trend of the metrics;
derive (i) one or more temporal properties, and (ii) one or more spatio-temporal properties of each of the plurality of metrics in accordance with the detected steady state;
generate (a) a static threshold, (b) a dynamic threshold using temporal analysis, and (c) a composite threshold based on the derived (i) the one or more temporal properties, and (ii) the one or more spatio-temporal properties of each of the plurality of metrics using one or more models, wherein the static threshold is generated based on a mean μ(g), a standard variation σ(g), and an aggressiveness factor α of each of the plurality of metrics related to the one or more entities in the system environment, wherein a value of the aggressiveness factor α is derived based on closeness of data distribution to a normal behavior, wherein when the metric under consideration observes variation, then the static threshold (ST) is defined as:
ST
(
upper
)
=
μ
(
g
)
+
ασ
(
g
)
ST
(
lower
)
=
μ
(
g
)
-
ασ
(
g
)
,
wherein when the static threshold values are derived using behavioral properties of metrics, a plurality of domain constraints including upper limit of thresholds (UL) and lower limit of thresholds (LL) are factored to recommend an optimal threshold value computed as:
ST
(
upper
)
=
max
(
ST
(
upper
)
,
LL
)
ST
(
upper
)
=
min
(
ST
(
lower
)
,
UL
)
,
wherein the dynamic threshold represents temporal threshold recommendation that is generated when a metric of the plurality of metrics under consideration observes significant variations across varied temporal dimensions and is derived and recommended when the metric under consideration observes significant variations across varied temporal dimensions and follows a specific pattern;
detect one or more anomalies in the received plurality of data when a value of the generated at least one of (a) the static threshold, (b) the dynamic threshold, and (c) the composite threshold is exceeded to a predefined value;
adaptively update the one or more models to the at least one of (i) the one or more temporal properties, and (ii) the one or more spatio-temporal properties of each of the plurality of metrics related to the one or more entities in the IT system environment based on at least one of (a) a self-learning mechanism, and (b) a self-tuning mechanism,
wherein in the self-learning mechanism, one or more parameters associated with the plurality of metrics, including a temporal pattern, a level of aggressiveness, an intra-bucket variation (ITA), an inter-bucket variation (ITR), and a score generated using ITA and ITR, in accordance with one or more inputs obtained as a feedback from one or more users are learnt and replicates the learning on each of the plurality of metrics related to the one or more entities in the IT system environment,
wherein in the self-tuning mechanism, group of similar metrics are identified, and corresponding set of representative metrics for a cluster of the set of similar metrics are identified, followed by statistics derived from a previous run of temporal analysis to combine with a statistics of a real time delta data; and
applying the adaptively updated models across domains to suppress false alerts and capture true positives.
8 . The system of claim 7 , further comprising:
continuously monitoring, a mismatch between the generated at least one of (a) the static threshold, (b) the dynamic threshold, and (c) the composite threshold with a pre-determined threshold; and
recommending, a corrected threshold when a mismatch between the generated at least one of (a) the static threshold, (b) the dynamic threshold, and (c) the composite threshold and the pre-determined threshold is detected.
9 . The system of claim 7 , wherein generating the dynamic threshold comprises:
resampling and categorizing the plurality of data into a plurality of buckets for a plurality of dimensions of the one or more temporal properties of each of the plurality of metrics related to the one or more entities in the system environment;
computing a representative value for each bucket from the plurality of buckets, wherein the representative value is computed based on a mean, and a standard deviation for each bucket from the plurality of buckets for each dimension from the plurality of dimensions;
computing an intra-bucket variation (ITA) for the plurality of data and inter-bucket variation (ITR) for the representative value for each bucket from the plurality of buckets, wherein the intra-bucket variation (ITA) captures average spread of a plurality of data points within each bucket from the plurality of buckets and the inter-bucket variation (ITR) captures variation of the plurality of data points across the plurality of dimensions;
generating a score for the one or more temporal properties of each of the plurality of metrics related to the one or more entities in the system environment based on the ITR and ITA;
identifying an optimal temporal property from the one or more temporal properties of each of the plurality of metrics related to the one or more entities in the system environment based on the generated score; and
generating the dynamic threshold based on the mean, the standard variation and an aggressiveness factor of each of the plurality of metrics related to the one or more entities in the system environment in accordance with the identified temporal property from the one or more temporal properties as
DT
i
(
upper
)
=
μ
(
b
)
i
+
α
σ
(
b
)
i
∀
i
∈
1
,
2
,
…
N
(
b
)
DT
i
(
lower
)
=
μ
(
b
)
i
-
α
σ
(
b
)
i
∀
i
∈
1
,
2
,
…
N
(
b
)
,
wherein when the dynamic threshold values are derived using behavioral properties of metrics, a plurality of domain constraints including an upper limit UL and a lower limit LL are factored to recommend an optimal threshold value for each dimension value computed as:
D
T
i
=
max
(
D
T
i
,
LL
)
∀
i
∈
1
,
2
,
…
N
(
b
)
D
T
i
=
min
(
D
T
i
,
UL
)
∀
i
∈
1
,
2
,
…
N
(
b
)
.
10 . The system of claim 7 , wherein generating the composite threshold comprises:
receiving, a first set of events corresponding to an entity identified from the one or more entities in the system environment with no metric;
mining, using one or more data sources, an influencing relationship between a node of the identified entity and a plurality of influencing metrics corresponding to the node of the identified entity to define a set of spatial dependencies;
applying a temporal threshold analysis to each influencing metric from the plurality of influencing metrics to generate at least one of a (i) a respective static threshold, (ii) a respective dynamic threshold, and (iii) a second set of events;
correlating the second set of events with the first set of events occurring in the node of the identified entity to access a plurality of temporal correlations; and
generating the composite threshold based on the plurality of temporal correlations, wherein the plurality of temporal correlations are indicative of behavior of the plurality of influencing metrics.
11 . The system of claim 7 , wherein the self-learning mechanism comprises:
generating, a plurality of clusters of the plurality of metrics related to the one or more entities based on a plurality of metric properties;
identifying a set of representative metrics for each cluster from the plurality of clusters of the plurality of metrics based on a similarity in the plurality of metric properties;
generating, at least one of (i) a plurality of optimal static thresholds, (ii) a plurality of optimal dynamic thresholds, and (iii) a plurality of optimal composite thresholds across varying degree of aggressiveness for each representative metric from the set of representative metrics for each cluster from the plurality of clusters of the plurality of metrics;
selecting, an optimal threshold from the at least one of (i) the plurality of optimal static thresholds, (ii) the plurality of optimal dynamic thresholds, and (iii) the plurality of optimal composite thresholds by applying one or more inputs obtained as feedback from one or more users includes user's preferred dimensions, user-defined threshold, and user's preferred aggressiveness factor, on each of the set of representative metrics for each cluster from the plurality of clusters of the plurality of metrics and replicate learning on each of the plurality of metrics related to the one or more entities in the system environment;
determining, a change in one or more parameters associated with the plurality of metrics related to the one or more entities in the IT system environment based on the one or more inputs obtained as the feedback from the one or more users on each cluster from the plurality of clusters of the plurality of metrics; and
self-learning, the one or more parameters associated with the plurality of metrics in accordance with the one or more inputs obtained as the feedback from the one or more users.
12 . The system of claim 7 , wherein the self-tuning mechanism comprises:
obtaining, a plurality of real time delta data corresponding to the one or more entities and a set of metrics related to the one or more entities in the system environment for the plurality of real time delta data;
performing temporal analysis on the set of metrics related to the one or more entities in the system environment for the plurality of real-time delta data, wherein the temporal analysis comprising:
(a) generating, a plurality of clusters of the set of metrics related to the one or more entities in a system environment for the plurality of real time delta data based on the plurality of metric properties;
(b) identifying a representative metric for each cluster from the plurality of clusters of the set of metrics based on a similarity in the plurality of metric properties; and
(c) generating, at least one of (i) a plurality of optimal static thresholds, (ii) a plurality of optimal dynamic thresholds, and (iii) a plurality of optimal composite thresholds across varying degree of aggressiveness for the representative metric for each cluster from the plurality of clusters of the set of metrics;
determining, a change in the one or more parameters associated with the set of metrics related to the one or more entities in the IT system environment for the plurality of real time delta data; and
self-tuning, by auto-optimizing the one or more parameters associated with the set of metrics based on (a) the self-learning of the one or more parameters associated with the plurality of metrics and (b) the temporal analysis performed on the set of metrics related to the one or more entities in the system environment for the plurality of real-time delta data.
13 . One or more non-transitory computer readable mediums comprising one or more instructions which when executed by one or more hardware processors cause:
receiving, a plurality of data corresponding to one or more entities in a IT system environment and a plurality of metrics related to the one or more entities, wherein the IT system environment represents represent a setup of a domain agnostic infrastructure of an organization, wherein the one or more entities represent components of the IT system environment, wherein plurality of metrics include a CPU utilization, a memory utilization, a disk utilization, and a process with a start time, an end time and a runtime in a batch system;
detecting a steady state of each of the plurality of metrics with a change-point detection referring to a persistent change in behavior of the plurality of metrics in accordance with internal infrastructural changes, and the change-point detection is performed, prior to analyzing data for recommending the optimal threshold recommendation, to derive recent steady state of the metrics under consideration, and an ensemble of change-point detection algorithms is used to detect changes in a mean, a variation, patterns, and a trend of the metrics;
deriving, (i) one or more temporal properties, and (ii) one or more spatio-temporal properties of each of the plurality of metrics in accordance with the detected steady state;
generating, (a) a static threshold, (b) a dynamic threshold using temporal analysis, and (c) a composite threshold based on the derived (i) the one or more temporal properties, and (ii) the one or more spatio-temporal properties of each of the plurality of metrics using one or more models, wherein the static threshold is generated based on a mean μ(g), a standard variation σ(g), and an aggressiveness factor α of each of the plurality of metrics related to the one or more entities in the system environment, wherein a value of the aggressiveness factor α is derived based on closeness of data distribution to a normal behavior, wherein when the metric under consideration observes variation, then the static threshold (ST) is defined as:
ST
(
upper
)
=
μ
(
g
)
+
ασ
(
g
)
ST
(
lower
)
=
μ
(
g
)
-
ασ
(
g
)
,
wherein when the static threshold values are derived using behavioral properties of metrics, a plurality of domain constraints including upper limit of thresholds (UL) and lower limit of thresholds (LL) are factored to recommend an optimal threshold value computed as:
ST
(
upper
)
=
max
(
ST
(
upper
)
,
LL
)
ST
(
upper
)
=
min
(
ST
(
lower
)
,
UL
)
,
wherein the dynamic threshold represents temporal threshold recommendation that is generated when a metric of the plurality of metrics under consideration observes significant variations across varied temporal dimensions and is derived and recommended when the metric under consideration observes significant variations across varied temporal dimensions and follows a specific pattern;
detecting, one or more anomalies in the received plurality of data when a value of the generated at least one of (a) the static threshold, (b) the dynamic threshold, and (c) the composite threshold is exceeded to a predefined value;
adaptively updating, the one or more models to the at least one of (i) the one or more temporal properties, and (ii) the one or more spatio-temporal properties of each of the plurality of metrics related to the one or more entities in the IT system environment based on at least one of (a) a self-learning mechanism, and (b) a self-tuning mechanism,
wherein in the self-learning mechanism, one or more parameters associated with the plurality of metrics, including a temporal pattern, a level of aggressiveness, an intra-bucket variation (ITA), an inter-bucket variation (ITR), and a score generated using ITA and ITR, in accordance with one or more inputs obtained as a feedback from one or more users are learnt and replicates the learning on each of the plurality of metrics related to the one or more entities in the IT system environment,
wherein in the self-tuning mechanism, group of similar metrics are identified, and corresponding set of representative metrics for a cluster of the set of similar metrics are identified, followed by statistics derived from a previous run of temporal analysis to combine with a statistics of a real time delta data; and
applying the adaptively updated models across domains to suppress false alerts and capture true positives.