Low-latency streaming analytics
Systems and methods are disclosed for implementing a low-latency data stream monitoring system. The data stream monitoring system may obtain raw data from a data source as soon after the data is generated, and may classify the data according to different topics. The topics may be published in a publish-subscribe messaging model, and data enrichment systems may subscribe to the topics to receive data for enrichment. The data enrichment systems may supplement or replace the raw data with additional information, and may further classify or reclassify the enriched data into different topics. The enriched data may then be published to an alert generation system, which may apply various criteria to the enriched data to determine that alerts should be generated, generate the alerts, and publish or transmit the alerts to client devices. Individual data streams, topics, enrichments, criteria, and alarms may be added, removed, or modified as required.
1. A method comprising:
iteratively processing a message through a multi-stage publish-subscribe messaging system, wherein the multi-stage publish-subscribe messaging system implements at least a first and second stage of processing at least partly in parallel, wherein implementing the first stage of processing includes:
publishing a modified message from the multi-stage publish-subscribe messaging system to the multi-stage publish-subscribe messaging system for continued processing at the second stage of processing, wherein the modified message is generated by modifying content of the message, and
wherein implementing the second stage of processing includes:
publishing an output based at least in part on evaluating the modified message according to a set of rules maintained by the multi-stage publish-subscribe messaging system.
2. The method of claim 1 , wherein the output comprises an alert that is generated based at least in part on a determination that a first rule of the set of rules has been satisfied.
3. The method of claim 1 , wherein evaluating the modified message according to the set of rules comprises applying at least a first rule of the set of rules to modified content of the modified message.
4. The method of claim 1 further comprising generating the modified message.
5. The method of claim 1 , wherein modifying the content of the message comprises replacing at least a portion of the content with data obtained from an external data source.
6. The method of claim 1 , wherein modifying the content of the message comprises enriching the content with data obtained from an external data source.
7. The method of claim 1 further comprising publishing the output from the multi-stage publish-subscribe messaging system to the multi-stage publish-subscribe messaging system for continued processing at a third stage of processing.
8. The method of claim 1 further comprising obtaining the set of rules based at least in part on modified content of the modified message.
9. The method of claim 1 further comprising associating a topic with the modified message based at least in part on modified content of the modified message.
10. The method of claim 1 , wherein the message is obtained by processing streaming data from a data source.
11. The method of claim 1 , wherein modifying the content of the message comprises publishing the message to a data enrichment system.
12. The method of claim 1 , wherein implementing the first stage of processing further includes determining, based at least in part on the content of the message, that data enrichment is available for the message.
13. The method of claim 1 , wherein implementing the first stage of processing further includes publishing the modified message to a subscriber.
14. The method of claim 1 , wherein the output includes at least a portion of the modified message.
15. The method of claim 1 further comprising identifying the second stage of processing based at least in part on the message.
16. The method of claim 1 further comprising identifying the second stage of processing based at least in part on the modified message.
17. A system comprising:
a data store including computer-executable instructions; and
a processor configured to execute the computer-executable instructions to:
iteratively process a message through a multi-stage publish-subscribe messaging system, wherein the multi-stage publish-subscribe messaging system implements at least a first and second stage of processing at least partly in parallel, wherein implementing the first stage of processing includes:
publishing a modified message from the multi-stage publish-subscribe messaging system to the multi-stage publish-subscribe messaging system for continued processing at the second stage of processing, wherein the modified message is generated by modifying content of the message, and
wherein implementing the second stage of processing includes:
publishing an output based at least in part on evaluating the modified message according to a set of rules maintained by the multi-stage publish-subscribe messaging system.
18. The system of claim 17 , wherein the computer-executable instructions further cause the processor to identify the set of rules based at least in part on the modified message.
19. One or more non-transitory computer-readable media comprising computer-executable instructions that, when executed by a computing system, cause the computing system to:
iteratively process a message through a multi-stage publish-subscribe messaging system, wherein the multi-stage publish-subscribe messaging system implements at least a first and second stage of processing at least partly in parallel, wherein implementing the first stage of processing includes:
publishing a modified message from the multi-stage publish-subscribe messaging system to the multi-stage publish-subscribe messaging system for continued processing at the second stage of processing, wherein the modified message is generated by modifying content of the message, and wherein implementing the second stage of processing includes:
publishing an output based at least in part on evaluating the modified message according to a set of rules maintained by the multi-stage publish-subscribe messaging system.
20. The one or more non-transitory computer-readable media of claim 19 , wherein the output comprises a further modified message, and wherein the further modified message is generated by modifying content of the modified message.