Time-based aggregation to feed a rete engine
Techniques to correlate event data are disclosed. In various embodiments, an aggregation engine is used to correlate event data received from one or more source systems based on one or more correlation rules. An event group comprising at least a portion of said correlated event data is generated programmatically and is asserted as a fact in a working memory of a Rete engine configured to apply one or more Rete rules to facts in the working memory.
1. A computer-implemented method of correlating streaming event data, comprising:
receiving event data from one or more source systems;
executing, using an aggregation engine, at least one aggregation policy programmatically based on one or more predefined correlation rules to:
generate event groups;
associate the event data with event groups, two or more events associated with an event group based on the one or more aggregation policies;
link a plurality of event data with a plurality of event groups based on the one or more aggregation policies;
create a temporary data structure for each event group;
populate the temporary data structure for each event group with event data, creating linked temporary data structures;
determine if the one or more aggregation policies are satisfied;
store the plurality of event groups as facts in a working memory of a Rete engine;
applying, using the Rete engine, Rete rules to the facts in the working memory, the Rete rules derived, at least in part, using the predefined correlation rules;
dynamically updating at least one selected from a group comprising the temporary data structure and the event groups in working memory based on a subsequently received event data;
wherein the predefined correlation rules comprises: at least one grouping clause, and at least one correlation constraint, at least one filtering clause, and at least one having clause;
wherein the at least one correlation constraint comprises a join constraint, sequencing constraint, and a free form constraint.
2. The method of claim 1 , wherein each of said source systems is configured to generate and provide to the aggregation engine discrete event data associated with that source system.
3. The method of claim 1 , wherein said source systems comprise an installed base of systems and said event data comprises system log data.
4. The method of claim 1 , wherein said correlation rules comprise user-defined rules.
5. The method of claim 1 , further comprising deriving one or more aggregation policies programmatically based at least in part on said correlation rules.
6. The method of claim 1 , wherein the event groups comprise a first event group and further comprising merging the first event group with a second event group.
7. The method of claim 6 , wherein the first event group is merged with the second event group based at least in part on a determination that a merger criterion is satisfied and the merger criterion is based on user-defined rules, wherein the first event group and the second event group are instantiated.
8. The method of claim 1 , wherein the aggregation engine is configured to extract one or more key-value pairs from a received event and to discard at least an unused portion of the received event.
9. The method of claim 1 , wherein the aggregation engine is configured to ignore and discard a received event that is determined to not be associated with an applicable correlation rule.
10. The method of claim 1 , wherein the aggregation engine is configured to associate a timestamp with an instance of received event data and to use the timestamp to apply one or more of said correlation rules, wherein the timestamp is used to determine how long to keep event data and/or to reorder events received out of order based on a rolling window size specified in applicable correlation rules.
11. The method of claim 10 , wherein the timestamp is used to implement a sliding window associated with one or more of said correlation rules.
12. The method of claim 1 , wherein a change in at least one rule of the Rete rules causes a change to at least one event group.
13. A system to correlate streaming event data, comprising:
a communication interface; and
a processor coupled to the communication interface and configured to:
receive event data from one or more source systems;
execute at least one aggregation policy programmatically based on one or more predefined correlation rules to:
generate event groups;
associate the event data with event groups, two or more events associated with an event group based on the one or more aggregation policies;
link a plurality of event data with a plurality of event groups based on the one or more aggregation policies;
create a temporary data structure for each event group;
populate the temporary data structure for each event group with event data, creating linked temporary data structures;
determine if the one or more aggregation policies are satisfied;
store the plurality of event groups as facts in a working memory of a Rete engine;
applying, using the Rete engine, Rete rules to the facts in the working memory, the Rete rules derived, at least in part, using the predefined correlation rules;
updating at least one selected from a group comprising the temporary data structure and the event groups in working memory based on a subsequently received event data;
wherein the predefined correlation rules comprises: at least one grouping clause, and at least one correlation constraint, at least one filtering clause, and at least one having clause;
wherein the at least one correlation constraint comprises a join constraint, sequencing constraint, and a free form constraint.
14. The system of claim 13 , wherein said correlation rules comprise user-defined rules.
15. The system of claim 13 , wherein the event groups comprise a first event group and further comprising merging the first event group with a second event group.
16. The system of claim 13 , wherein the processor is configured to extract one or more key-value pairs from a received event and to discard at least an unused portion of the received event.
17. The system of claim 13 , wherein the processor is configured to ignore and discard a received event that is determined to not be associated with an applicable correlation rule.
18. The system of claim 13 , wherein a change in at least one rule of the Rete rules causes a change to at least one event group.
19. A computer program product to correlate streaming event data, the computer program product being embodied in a non-transitory computer readable medium and comprising computer instructions for:
receiving event data from one or more source systems;
executing at least one aggregation policy programmatically based on one or more predefined correlation rules to:
generate event groups;
associate the event data with event groups, two or more events associated with an event group based on the one or more aggregation policies;
link a plurality of event data with a plurality of event groups based on the one or more aggregation policies;
create a temporary data structure for each event;
populate the temporary data structure for each event group with event data, creating linked temporary data structures;
determine if the one or more aggregation policies are satisfied;
store the plurality of event groups as facts in a working memory of a Rete engine;
applying, using the Rete engine, Rete rules to the facts in the working memory, the Rete rules derived, at least in part, using the predefined correlation rules;
dynamically updating at least one selected from a group comprising the temporary data structure and the event groups in working memory based on a subsequently received event data;
wherein the predefined correlation rules comprises: at least one grouping clause, and at least one correlation constraint, at least one filtering clause, and at least one having clause;
wherein the at least one correlation constraint comprises a loin constraint, sequencing constraint, and a free form constraint.
20. The computer program product of claim 19 , wherein a change in at least one rule of the Rete rules causes a change to at least one event group.