Anomaly detection through clustering of time-series data subsequences and determination of adaptive thresholding
Computerized methodologies are disclosed that are directed to detecting anomalies within a time-series data set. An aspect of the anomaly detection process includes determining one or more seasonality patterns that correspond to a specific time-series data set by evaluating a set of candidate seasonality patterns (e.g., hourly, daily, weekly, day-start off-sets, etc.). The evaluation of a candidate seasonality pattern may include dividing the time-series data set into a collection of subsequences based on the particular candidate seasonality pattern. Further, the collection of subsequences may be divided into clusters and a silhouette score may be computed to measure the clustering quality of the candidate seasonality pattern. In some instances, the candidate seasonality pattern having the highest silhouette score is selected and utilized in anomaly detection process. In other instances, a plurality of seasonality patterns may be combined forming a time policy, where the time policy is utilized in anomaly detection process.
1. A computer-implemented method, comprising:
obtaining a time-series data set;
performing a data regularity check on the time-series data set;
responsive to the data regularity check not satisfying a data regularity threshold, performing a data aggregation process to regularize the time-series data set;
dividing the time-series data set into a set of subsequences based on a first candidate seasonality pattern;
clustering the set of subsequences in accordance with the first candidate seasonality pattern;
determining a silhouette score measuring a quality of the clustering, wherein the silhouette score indicates how well the first candidate seasonality pattern fits the time-series data set;
establishing an anomaly band for the time-series data set based on the first candidate seasonality pattern when the silhouette score satisfies a threshold comparison; and
detecting one or more anomalies within the time-series data set that are outside of the anomaly band.
2. The method of claim 1 , wherein the first candidate seasonality pattern is a set of parameters corresponding to values of data points comprising the time-series data set indicating an expected pattern of the values of the data points.
3. The method of claim 1 , wherein the first candidate seasonality pattern defines a methodology for dividing the time-series data set based on one or more of time of day or day of week.
4. The method of claim 1 , wherein computing the silhouette score for the first candidate seasonality pattern includes (i) computing a silhouette score for data points of the time-series data set, and (ii) determining a mean or a medium of the silhouette scores for the data points.
5. The method of claim 4 , wherein computing the silhouette score for the first data point of the data points comprising the time-series data set includes:
determining, for each cluster, an average distance between the first data point and data points belonging to clusters to which the first data point does not belong,
dividing (a) a difference between (i) a minimum of the average distances between the first data point and the data points belonging to the clusters to which the first data point does not belong, and (ii) an average distance between the first data point and the other data points belonging to a cluster to which the first data point does belong, by (b) a maximum of (i) the minimum of the average distances between the first data point and the data points belonging to the clusters to which the first data point does not belong, and (ii) the average distance between the first data point and the other data points belonging to the cluster to which the first data point does belong.
6. The method of claim 1 , wherein the anomaly band includes an upper threshold and a lower threshold relative to the time-series data set.
7. The method of claim 1 , wherein the time-series data set includes machine generated data from within a networked environment and comprises a set of data points each associated with a timestamp.
8. A computing device, comprising:
a processor; and
a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations including:
obtaining a time-series data set;
performing a data regularity check on the time-series data set;
responsive to the data regularity check not satisfying a data regularity threshold, performing a data aggregation process to regularize the time-series data set;
dividing the time-series data set into a set of subsequences based on a first candidate seasonality pattern;
clustering the set of subsequences in accordance with the first candidate seasonality pattern;
determining a silhouette score measuring a quality of the clustering, wherein the silhouette score indicates how well the first candidate seasonality pattern fits the time-series data set;
establishing an anomaly band for the time-series data set based on the first candidate seasonality pattern when the silhouette score satisfies a threshold comparison; and
detecting one or more anomalies within the time-series data set that are outside of the anomaly band.
9. The computing device of claim 8 , wherein the first candidate seasonality pattern is a set of parameters corresponding to values of data points comprising the time-series data set indicating an expected pattern of the values of the data points.
10. The computing device of claim 8 , wherein the first candidate seasonality pattern defines a methodology for dividing the time-series data set based on one or more of time of day or day of week.
11. The computing device of claim 8 , wherein computing the silhouette score for the first candidate seasonality pattern includes (i) computing a silhouette score for data points of the time-series data set, and (ii) determining a mean or a medium of the silhouette scores for the data points.
12. The computing device of claim 11 , and wherein computing a silhouette score for a first data point of the data points comprising the time-series data set includes:
determining, for each cluster, an average distance between the first data point and data points belonging to clusters to which the first data point does not belong,
dividing (a) a difference between (i) a minimum of the average distances between the first data point and the data points belonging to the clusters to which the first data point does not belong, and (ii) an average distance between the first data point and the other data points belonging to a cluster to which the first data point does belong, by (b) a maximum of (i) the minimum of the average distances between the first data point and the data points belonging to the clusters to which the first data point does not belong, and (ii) the average distance between the first data point and the other data points belonging to the cluster to which the first data point does belong.
13. The computing device of claim 8 , wherein the anomaly band includes an upper threshold and a lower threshold relative to the time-series data set.
14. The computing device of claim 8 , wherein the time-series data set includes machine generated data from within a networked environment and comprises a set of data points each associated with a timestamp.
15. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processor to perform operations including:
obtaining a time-series data set;
performing a data regularity check on the time-series data set;
responsive to the data regularity check not satisfying a data regularity threshold, performing a data aggregation process to regularize the time-series data set;
dividing the time-series data set into a set of subsequences based on a first candidate seasonality pattern;
clustering the set of subsequences in accordance with the first candidate seasonality pattern;
determining a silhouette score measuring a quality of the clustering, wherein the silhouette score indicates how well the first candidate seasonality pattern fits the time-series data set;
establishing an anomaly band for the time-series data set based on the first candidate seasonality pattern when the silhouette score satisfies a threshold comparison; and
detecting one or more anomalies within the time-series data set that are outside of the anomaly band.
16. The non-transitory computer-readable medium of claim 15 , wherein the first candidate seasonality pattern is a set of parameters corresponding to values of data points comprising the time-series data set indicating an expected pattern of the values of the data points.
17. The non-transitory computer-readable medium of claim 15 , wherein the first candidate seasonality pattern defines a methodology for dividing the time-series data set based on one or more of time of day or day of week.
18. The non-transitory computer-readable medium of claim 15 , wherein computing the silhouette score for the first candidate seasonality pattern includes (i) computing a silhouette score for data points of the time-series data set, and (ii) determining a mean or a medium of the silhouette scores for the data points; and
wherein computing the silhouette score for the first data point of the data points comprising the time-series data set includes:
determining, for each cluster, an average distance between the first data point and data points belonging to clusters to which the first data point does not belong,
dividing (a) a difference between (i) a minimum of the average distances between the first data point and the data points belonging to the clusters to which the first data point does not belong, and (ii) an average distance between the first data point and the other data points belonging to a cluster to which the first data point does belong, by (b) a maximum of (i) the minimum of the average distances between the first data point and the data points belonging to the clusters to which the first data point does not belong, and (ii) the average distance between the first data point and the other data points belonging to the cluster to which the first data point does belong.
19. The non-transitory computer-readable medium of claim 15 , wherein the anomaly band includes an upper threshold and a lower threshold relative to the time-series data set.
20. The non-transitory computer-readable medium of claim 15 , wherein the time-series data set includes machine generated data from within a networked environment and comprises a set of data points each associated with a timestamp.