Anomaly detection for data stream processing
A continuous anomaly detection service receives data stream and performs continuous anomaly detection on the incoming data streams. This continuous anomaly detection is performed based on anomaly detection definitions, which define a signal used for anomaly detection and an anomaly detection configuration. These anomaly detection definitions can be modified, such that continuous anomaly detection continues to be performed for the data stream and the signal, based on the new anomaly detection definition.
1. A computer-implemented method comprising:
accessing an anomaly detection definition that defines how to populate a buffered signal of a sequential set of time-series data points;
determining that one of a plurality of data streams is associated with the anomaly detection definition; and
upon receiving an updated data point from the data stream, performing anomaly detection by:
inserting the updated data point from the data stream into the buffered signal of the sequential set of time-series data points; and
analyzing the buffered signal of the sequential set of time-series data points to determine a corresponding anomaly result for the anomaly detection definition.
2. The computer-implemented method of claim 1 , further comprising receiving the plurality of data streams as real-time data provided from data sources from an information technology system.
3. The computer-implemented method of claim 1 , further comprising receiving data from the data stream in an order that the data was created.
4. The computer-implemented method of claim 1 , further comprising processing the data stream by indexing incoming data from the data stream and generating the updated data point as a key performance indicator from the incoming data.
5. The computer-implemented method of claim 1 , further comprising applying an anomaly detection search command to the plurality of data streams to extract the updated data point for the data stream.
6. The computer-implemented method of claim 1 , further comprising updating the buffered signal of the sequential set of time-series data points with data points from different data sources arriving at different intervals.
7. The computer-implemented method of claim 1 , further comprising receiving the plurality of data streams from a plurality of data sources, each of the plurality of data sources providing multiple data streams of the plurality of data streams.
8. The computer-implemented method of claim 1 , further comprising receiving multiple data streams of the plurality of data streams from a single data source, wherein each of the multiple data streams reflects a different aspect of data provided by the single data source.
9. The computer-implemented method of claim 1 , wherein the anomaly detection definition is one of a plurality of anomaly detection definitions, and wherein a configuration object includes all of the plurality of anomaly detection definitions.
10. The computer-implemented method of claim 1 , wherein the anomaly detection definition is one of a plurality of anomaly detection definitions, wherein a configuration object includes all of the plurality of anomaly detection definitions, and wherein the configuration object defines a shared search for the plurality of anomaly detection definitions.
11. A non-transitory computer-readable storage medium comprising instructions stored thereon which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
accessing an anomaly detection definition that defines how to populate a buffered signal of a sequential set of time-series data points;
determining that one of a plurality of data streams is associated with the anomaly detection definition; and
upon receiving an updated data point from the data stream, performing anomaly detection by:
inserting the updated data point from the data stream into the buffered signal of the sequential set of time-series data points; and
analyzing the buffered signal of the sequential set of time-series data points to determine a corresponding anomaly result for the anomaly detection definition.
12. The non-transitory computer-readable storage medium of claim 11 , the operations further comprising receiving the plurality of data streams as real-time data provided from data sources from an information technology system.
13. The non-transitory computer-readable storage medium of claim 11 , the operations further comprising receiving data from the data stream in an order that the data was created.
14. The non-transitory computer-readable storage medium of claim 11 , the operations further comprising processing the data stream by indexing incoming data from the data stream and generating the updated data point as a key performance indicator from the incoming data.
15. The non-transitory computer-readable storage medium of claim 11 , the operations further comprising applying an anomaly detection search command to the plurality of data streams to extract the updated data point for the data stream.
16. The non-transitory computer-readable storage medium of claim 11 , the operations further comprising updating the buffered signal of the sequential set of time-series data points with data points from different data sources arriving at different intervals.
17. The non-transitory computer-readable storage medium of claim 11 , the operations further comprising receiving the plurality of data streams from a plurality of data sources, each of the plurality of data sources providing multiple data streams of the plurality of data streams.
18. The non-transitory computer-readable storage medium of claim 11 , the operations further comprising receiving multiple data streams of the plurality of data streams from a single data source, wherein each of the multiple data streams reflects a different aspect of data provided by the single data source.
19. The non-transitory computer-readable storage medium of claim 11 , wherein the anomaly detection definition is one of a plurality of anomaly detection definitions, and wherein a configuration object includes all of the plurality of anomaly detection definitions.
20. The non-transitory computer-readable storage medium of claim 11 , wherein the anomaly detection definition is one of a plurality of anomaly detection definitions, wherein a configuration object includes all of the plurality of anomaly detection definitions, and wherein the configuration object defines a shared search for the plurality of anomaly detection definitions.
21. A system for performing anomaly detection, comprising:
at least one processor; and
at least one memory having instructions stored thereon which, when executed by the at least one processor, cause the at least one processor to perform operations comprising
accessing an anomaly detection definition that defines how to populate a buffered signal of a sequential set of time-series data points;
determining that one of a plurality of data streams is associated with the anomaly detection definition; and
upon receiving an updated data point from the data stream, performing anomaly detection by:
inserting the updated data point from the data stream into the buffered signal of the sequential set of time-series data points; and
analyzing the buffered signal of the sequential set of time-series data points to determine a corresponding anomaly result for the anomaly detection definition.
22. The system of claim 21 , the operations further comprising receiving the plurality of data streams as real-time data provided from data sources from an information technology system.
23. The system of claim 21 , the operations further comprising receiving data from the data stream in an order that the data was created.
24. The system of claim 21 , the operations further comprising processing the data stream by indexing incoming data from the data stream and generating the updated data point as a key performance indicator from the incoming data.
25. The system of claim 21 , the operations further comprising applying an anomaly detection search command to the plurality of data streams to extract the updated data point for the data stream.
26. The system of claim 21 , the operations further comprising updating the buffered signal of the sequential set of time-series data points with data points from different data sources arriving at different intervals.
27. The system of claim 21 , the operations further comprising receiving the plurality of data streams from a plurality of data sources, each of the plurality of data sources providing multiple data streams of the plurality of data streams.
28. The system of claim 21 , the operations further comprising receiving multiple data streams of the plurality of data streams from a single data source, wherein each of the multiple data streams reflects a different aspect of data provided by the single data source.
29. The system of claim 21 , wherein the anomaly detection definition is one of a plurality of anomaly detection definitions, and wherein a configuration object includes all of the plurality of anomaly detection definitions.
30. The system of claim 21 , wherein the anomaly detection definition is one of a plurality of anomaly detection definitions, wherein a configuration object that includes all of the plurality of anomaly detection definitions, and wherein the configuration object defines a shared search for the plurality of anomaly detection definitions.