Adaptive thresholding of data streamed to a data processing pipeline
Systems and methods are described for performing adaptive thresholding on key performance indicator (KPI) values using an online machine learning algorithm as the KPI values or the data from which the KPI values are derived is being ingested. For example, the system can identify outliers in a moving window of KPI values. To implement the adaptive thresholding, the system may identify seasonality and/or trend components in historical KPI values. When a new KPI value is obtained, the system may remove the identified seasonality and/or trend components from the KPI value, and determine whether the modified KPI value is an outlier using sketches or quantiles. The system can then repeat this process for each subsequently received KPI value.
1. A computer-implemented method comprising:
receiving historical key performance indicator (KPI) data corresponding to a period of time;
determining, at a first time, a seasonal component of the historical KPI data;
obtaining, by a data processing component of a data processing pipeline at a second time after the first time, a KPI value on which to perform adaptive thresholding;
decomposing the KPI value based at least in part on the determined seasonal component to form a modified version of the KPI value;
generating a sketch for the modified version of the KPI value;
determining a quantile value based at least in part on the sketch and a second sketch generated for a modified version of a second KPI value obtained by the data processing component at a third time before the second time;
determining that the modified version of the KPI value exceeds the quantile value; and
outputting an indication that the KPI value is an outlier in response to the determination that the modified version of the KPI value exceeds the quantile value.
2. The method of claim 1 , further comprising obtaining a stream of KPI values in sequence, wherein the stream of KPI values comprises the KPI value.
3. The method of claim 1 , further comprising obtaining a stream of KPI values in sequence after obtaining the KPI value.
4. The method of claim 1 , further comprising:
removing the seasonal component from the historical KPI data to form modified historical KPI data;
performing adaptive thresholding on the modified historical KPI data using a batch process; and
causing a preview of one or more values of the modified historical KPI data to be displayed, wherein the preview comprises an indication of a value in the one or more values of the modified historical KPI data identified as being an outlier based on the adaptive thresholding being performed.
5. The method of claim 1 , further comprising:
removing the seasonal component from the historical KPI data to form modified historical KPI data;
performing adaptive thresholding on the modified historical KPI data using a batch process;
causing a preview of one or more values of the modified historical KPI data to be displayed, wherein the preview comprises an indication of a value in the one or more values of the modified historical KPI data identified as being an outlier based on the adaptive thresholding being performed;
in response to causing the preview to be displayed, receiving an indication of a requested adjustment to the performance of the adaptive thresholding;
performing adaptive thresholding on the modified historical KPI data using the requested adjustment; and
causing a second preview of the one or more values of the modified historical KPI data to be displayed, wherein the second preview is generated in response to performing the adaptive thresholding on the modified historical KPI data using the requested adjustment.
6. The method of claim 1 , further comprising:
obtaining, at a fourth time after outputting the indication that the KPI value is an outlier, the historical KPI data and the KPI value; and
determining a second seasonal component using at least some of the historical KPI data and the KPI value.
7. The method of claim 1 , further comprising:
obtaining, at a fourth time after outputting the indication that the KPI value is an outlier, the historical KPI data and the KPI value;
determining a second seasonal component using at least some of the historical KPI data and the KPI value;
obtaining a third KPI value;
decomposing the third KPI value based at least in part on the determined second seasonal component to form a modified version of the third KPI value; and
performing adaptive thresholding on the modified version of the third KPI value.
8. The method of claim 1 , further comprising:
ingesting, by the data processing pipeline, raw machine data from a data stream source; and
determining, by the data processing pipeline, the KPI value using the raw machine data.
9. The method of claim 1 , further comprising:
ingesting, by the data processing pipeline, raw machine data from a data stream source;
running, by the data processing pipeline, a search query on the raw machine data; and
determining the KPI value based on results of the search query.
10. The method of claim 1 , wherein decomposing the KPI value further comprises removing the determined seasonal component from the KPI value to form the modified version of the KPI value.
11. The method of claim 1 , wherein decomposing the KPI value further comprises removing the determined seasonal component and a trend component from the KPI value to form the modified version of the KPI value.
12. The method of claim 1 , wherein determining a quantile value based at least in part on the sketch and a second sketch generated for a modified version of a second KPI value obtained by the data processing component at a third time before the second time further comprises:
merging the sketch and the second sketch to form a merged sketch; and
determining the quantile value based at least in part on the merged sketch.
13. The method of claim 1 , wherein determining a quantile value based at least in part on the sketch and a second sketch generated for a modified version of a second KPI value obtained by the data processing component at a third time before the second time further comprises:
determining that a third sketch includes a modified version of a third KPI value having a timestamp that falls outside of a time window;
discarding the third sketch; and
determining the quantile value based on the sketch and the second sketch and not on the third sketch.
14. The method of claim 1 , wherein the sketch comprises a Gaussian sketch.
15. The method of claim 1 , wherein the KPI value is a value of at least one parameter of an entity that provides a service.
16. The method of claim 1 , wherein the KPI value is a value of at least one operational parameter of a computing system.
17. The method of claim 1 , wherein outputting an indication that the KPI value is an outlier further comprises causing a graphical representation of the KPI value and a second KPI value to be displayed, wherein the graphical representation comprises an annotation that indicates that the KPI value is the outlier.
18. The method of claim 1 , wherein outputting an indication that the KPI value is an outlier further comprises causing an electronic message to be transmitted that lists the KPI value as being the outlier.
19. A system comprising:
a data store including computer-executable instructions; and
one or more processors configured to execute the computer-executable instructions, wherein execution of the computer-executable instructions causes the system to:
receive historical key performance indicator (KPI) data corresponding to a period of time;
determine, at a first time, a seasonal component of the historical KPI data;
obtain, at a second time after the first time, a KPI value on which to perform adaptive thresholding;
decompose the KPI value based at least in part on the determined seasonal component to form a modified version of the KPI value;
generate a sketch for the modified version of the KPI value;
determine a quantile value based at least in part on the sketch and a second sketch generated for a modified version of a second KPI value obtained by a data processing node at a third time before the second time;
determine that the modified version of the KPI value exceeds the quantile value; and
output an indication that the KPI value is an outlier in response to the determination that the modified version of the KPI value exceeds the quantile value.
20. The system of claim 19 , wherein execution of the computer-executable instructions further causes the system to:
remove the seasonal component from the historical KPI data to form modified historical KPI data;
perform adaptive thresholding on the modified historical KPI data using a batch process; and
cause a preview of one or more values of the modified historical KPI data to be displayed, wherein the preview comprises an indication of a value in the one or more values of the modified historical KPI data identified as being an outlier based on the adaptive thresholding being performed.
21. Non-transitory computer-readable media including computer-executable instructions that, when executed by a computing system, cause the computing system to:
receive historical key performance indicator (KPI) data corresponding to a period of time;
determine, at a first time, a seasonal component of the historical KPI data;
obtain, at a second time after the first time, a KPI value on which to perform adaptive thresholding;
decompose the KPI value based at least in part on the determined seasonal component to form a modified version of the KPI value;
generate a sketch for the modified version of the KPI value;
determine a quantile value based at least in part on the sketch and a second sketch generated for a modified version of a second KPI value obtained by a data processing node at a third time before the second time;
determine that the modified version of the KPI value exceeds the quantile value; and
output an indication that the KPI value is an outlier in response to the determination that the modified version of the KPI value exceeds the quantile value.
22. The non-transitory computer-readable media of claim 21 , wherein the computer-executable instructions, when executed by the computing system, further cause the computing system to:
remove the seasonal component from the historical KPI data to form modified historical KPI data;
perform adaptive thresholding on the modified historical KPI data using a batch process; and
cause a preview of one or more values of the modified historical KPI data to be displayed, wherein the preview comprises an indication of a value in the one or more values of the modified historical KPI data identified as being an outlier based on the adaptive thresholding being performed.