Anomaly detection from incoming data from a data stream
A continuous anomaly detection service receives data stream and performs continuous anomaly detection on the incoming data streams. This continuous anomaly detection is performed based on anomaly detection definitions, which define a signal used for anomaly detection and an anomaly detection configuration. These anomaly detection definitions can be modified, such that continuous anomaly detection continues to be performed for the data stream and the signal, based on the new anomaly detection definition.
1. A computer-implemented method comprising:
extracting, from a chunk of incoming data from a data stream, one or more signal chunks;
generating one or more time-stamped data points from the data stream based on the one or more signal chunks;
inserting the one or more time-stamped data points from the data stream into a signal buffer storing a sequential set of time-series data points for an anomaly detection definition; and
analyzing the sequential set of time-series data points from the signal buffer to determine a corresponding anomaly result for the anomaly detection definition.
2. The computer-implemented method of claim 1 , wherein the extracting of the one or more signal chunks from the chunk of incoming data is by a router in communication with a definition manager that controls how the router routes data based on an anomaly detection configuration for the anomaly detection definition.
3. The computer-implemented method of claim 1 , further comprising extracting, from the chunk of incoming data, different signal chunks for different anomaly detection definitions and routing the different signal chunks to different signal buffers for the different anomaly detection definitions.
4. The computer-implemented method of claim 1 , further comprising issuing a backfill request based on a determination that a minimum required number of data points is not present.
5. The computer-implemented method of claim 1 , wherein the extracting of the one or more signal chunks from the chunk of incoming data is by a router, wherein a definition manager is configured to issue chunk routing requests for the router to route the one or more signal chunks extracted from the chunk of incoming data to the signal buffer, and to issue backfill requests for a backfill manager to retrieve and route one or more historical data points to the signal buffer.
6. The computer-implemented method of claim 1 , wherein the generating of the one or more time-stamped data points from the data stream comprises routing the one or more signal chunks to a reorder buffer associated with the signal buffer and reordering the one or more signal chunks in the reorder buffer based on timestamps extracted for the one or more signal chunks from the chunk of incoming data.
7. The computer-implemented method of claim 1 , further comprising pre-loading or backfilling the signal buffer with one or more historical data points based on a determination that a minimum required number of data points is not present, and ordering the one or more historical data points and the one or more time-stamped data points in the signal buffer in sequential order.
8. A non-transitory computer-readable storage medium comprising instructions stored thereon which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
extracting, from a chunk of incoming data from a data stream, one or more signal chunks;
generating one or more time-stamped data points from the data stream based on the one or more signal chunks;
inserting the one or more time-stamped data points from the data stream into a signal buffer storing a sequential set of time-series data points for an anomaly detection definition; and
analyzing the sequential set of time-series data points from the signal buffer to determine a corresponding anomaly result for the anomaly detection definition.
9. The non-transitory computer-readable storage medium of claim 8 , wherein the extracting of the one or more signal chunks from the chunk of incoming data is by a router in communication with a definition manager that controls how the router routes data based on an anomaly detection configuration for the anomaly detection definition.
10. The non-transitory computer-readable storage medium of claim 8 , the operations further comprising extracting, from the chunk of incoming data, different signal chunks for different anomaly detection definitions and routing the different signal chunks to different signal buffers for the different anomaly detection definitions.
11. The non-transitory computer-readable storage medium of claim 8 , the operations further comprising issuing a backfill request based on a determination that a minimum required number of data points is not present.
12. The non-transitory computer-readable storage medium of claim 8 , wherein the extracting of the one or more signal chunks from the chunk of incoming data is by a router, wherein a definition manager is configured to issue chunk routing requests for the router to route the one or more signal chunks extracted from the chunk of incoming data to the signal buffer, and to issue backfill requests for a backfill manager to retrieve and route one or more historical data points to the signal buffer.
13. The non-transitory computer-readable storage medium of claim 8 , wherein the generating of the one or more time-stamped data points from the data stream comprises routing the one or more signal chunks to a reorder buffer associated with the signal buffer and reordering the one or more signal chunks in the reorder buffer based on timestamps extracted for the one or more signal chunks from the chunk of incoming data.
14. The non-transitory computer-readable storage medium of claim 8 , the operations further comprising pre-loading or backfilling the signal buffer with one or more historical data points based on a determination that a minimum required number of data points is not present, and ordering the one or more historical data points and the one or more time-stamped data points in the signal buffer in sequential order.
15. A system comprising:
at least one processor; and
at least one memory having instructions stored thereon which, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
extracting, from a chunk of incoming data from a data stream, one or more signal chunks;
generating one or more time-stamped data points from the data stream based on the one or more signal chunks;
inserting the one or more time-stamped data points from the data stream into a signal buffer storing a sequential set of time-series data points for an anomaly detection definition; and
analyzing the sequential set of time-series data points from the signal buffer to determine a corresponding anomaly result for the anomaly detection definition.
16. The system of claim 15 , wherein the extracting of the one or more signal chunks from the chunk of incoming data is by a router in communication with a definition manager that controls how the router routes data based on an anomaly detection configuration for the anomaly detection definition.
17. The system of claim 15 , the operations further comprising extracting, from the chunk of incoming data, different signal chunks for different anomaly detection definitions and routing the different signal chunks to different signal buffers for the different anomaly detection definitions.
18. The system of claim 15 , the operations further comprising issuing a backfill request based on a determination that a minimum required number of data points is not present.
19. The system of claim 15 , wherein the extracting of the one or more signal chunks from the chunk of incoming data is by a router, wherein a definition manager is configured to issue chunk routing requests for the router to route the one or more signal chunks extracted from the chunk of incoming data to the signal buffer, and to issue backfill requests for a backfill manager to retrieve and route one or more historical data points to the signal buffer.
20. The system of claim 15 , wherein the generating of the one or more time-stamped data points from the data stream comprises routing the one or more signal chunks to a reorder buffer associated with the signal buffer and reordering the one or more signal chunks in the reorder buffer based on timestamps extracted for the one or more signal chunks from the chunk of incoming data.