Distributed event correlation system
View Patent ↗According to an example, a master node is to divide an event field in events into partitions including ordered contiguous blocks of values for the event field. Each partition may be assigned to a pair of cluster nodes. A partition map is determined from the partitions and may identify for each partition, the block of the event field values for the partition, a primary cluster node, and a failover cluster node for the primary cluster node.
1. A master node in a distributed event correlation system including cluster nodes in a cluster, the master node comprising:
at least one processor to:
divide an event field in events into partitions including ordered contiguous blocks of values for the event field;
assign each partition to a pair of the cluster nodes;
determine a partition map from the partitions, wherein the partition map identifies for each partition, the block of the event field values for the partition, a primary cluster node, and a failover cluster node for the primary cluster node; and
a data storage to store the partition map, wherein the at least one processor is to store the partition map in the data storage.
2. The master node of claim 1 , wherein the at least one processor is to:
receive an event from a data source;
extract a value for the event field from the event;
determine whether the extracted value is a non-null value;
if the extracted value is non-null, identify a corresponding cluster node for the received event based on the block in which the extracted value is located; and
send the event to the corresponding cluster node.
3. The master node of claim 2 , wherein the at least one processor is to:
if the extracted value is null, send the event to an independent cluster node, wherein the independent cluster node is not assigned to any of the contiguous blocks of values for the event field and is to correlate received events according an event rule.
4. The master node of claim 1 , wherein the at least one processor is to:
determine whether a new cluster node is joining the cluster;
if the new cluster node is joining the cluster, assign a portion of the contiguous blocks of event field values to the new cluster node; and
update the partition map to include the new cluster node and its partition and its failover cluster node and to include any changes to the partitions as a result of the assigning of the portion of the contiguous blocks of event field values to the new cluster node.
5. The master node of claim 4 , wherein the new cluster node loads application data for its partition from an event correlation system data storage storing events for the distributed event correlation system.
6. The master node of claim 1 , wherein the at least one processor is to:
determine whether a primary cluster node leaves the cluster;
if a primary cluster node leaves the cluster, send a message to the failover cluster node for the primary cluster node that left the cluster to become a new primary cluster node for the partition of the primary cluster node; and
update the partition map to include the failover cluster node as the new primary cluster node for the partition and to include a new failover cluster node for the new primary cluster node.
7. The master node of claim 6 , wherein the failover cluster node restores an aggregation state from a last received checkpoint, and loads application data for the partition from an event correlation system data storage storing application data for the distributed event correlation system.
8. The master node of claim 1 , wherein the at least one processor is to:
distribute the partition map to the cluster nodes;
update the partition map in response to changes in the cluster; and
distribute the updated partition map to the cluster nodes.
9. A cluster node of a plurality of cluster nodes in a distributed event correlation system storing partitions of event data, the cluster node comprising:
data storage to store events received for a partition for the cluster node; and
at least one processor to
receive a rule for correlating events, wherein events are to be aggregated to determine whether a condition in the rule is satisfied;
receive an event for the partition;
determine whether the event is a partial match for the rule, wherein the partial match includes a partial satisfaction of the condition;
if the event is a partial match, determines whether the rule is a partition-independent rule or a partitioned rule;
if the rule is a partition-independent rule, send information for the partial match to a rule designated node to aggregate events to determine whether the condition is satisfied; and
if the rule is a partitioned rule, the event is stored at the cluster node as a partial match and events are aggregated at the cluster node to determine whether the condition is satisfied.
10. The cluster node of claim 9 , wherein if the rule is the partition-independent rule, the rule designated node receives information for partial matches from multiple cluster nodes to aggregate events to determine whether the condition is satisfied.
11. The cluster node of claim 9 , wherein the at least one processor is to:
store and update an aggregation state, wherein the aggregation state includes information for the partial matches for the rule; and
send a checkpoint to a failover node for the cluster node, wherein the checkpoint includes the aggregation state and information for events recently received for the partition of the cluster node.
12. The cluster node of claim 9 , wherein the at least one processor is to receive a partition map determined by a master node in the distributed event correlation system and store the partition map in the data storage, wherein the master node is to divide an event field in events into the partitions including ordered contiguous blocks of values for an event field, and the partition map identifies for each partition, a block of event field values for the partition, a primary cluster node, and a failover cluster node for the primary cluster node.
13. The cluster node of claim 12 , wherein the cluster node is to receive incoming events for the partition from the master node, wherein the master node, for each incoming event is to extract a value for the event field from the event, identify a corresponding cluster node for the received event based on the block in which the extracted value is located; and send the event to the corresponding cluster node.
14. A non-transitory computer readable medium including machine readable instructions executable by at least one processor to:
divide an event field in events into partitions including ordered contiguous blocks of values for the event field;
assign each partition to a pair of cluster nodes in a distributed event correlation system;
determine a partition map from the partitions, wherein the partition map identifies for each partition, the block of the event field values for the partition, a primary cluster node, and a failover cluster node for the primary cluster node; and
store the partition map,
wherein each of the cluster nodes are to correlate events received from the master node based on a rule, where the correlation includes:
receiving an event for the partition;
determining whether the event is a partial match for the rule, wherein the partial match includes a partial satisfaction of the condition;
if the event is a partial match, determining whether the rule is a partition-independent rule or a partitioned rule;
if the rule is a partition-independent rule, sending information for the partial match to a rule designated node to aggregate events to determine whether the condition is satisfied; and
if the rule is a partitioned rule, storing the event at the cluster node as a partial match and aggregating events at the cluster node to determine whether the condition is satisfied.
15. A master node in a distributed event correlation system including cluster nodes in a cluster, the master node comprising:
at least one processor to:
divide an event field in events into partitions including ordered contiguous blocks of values for the event field;
assign each partition to a pair of the cluster nodes;
determine a partition map from the partitions, wherein the partition map identifies for each partition, the block of the event field values for the partition, a primary cluster node, and a failover cluster node for the primary cluster node; and
a data storage to store the partition map, wherein the at least one processor is to store the partition map in the data storage,
wherein each of the cluster nodes are to correlate events received from the master node based on a rule, where the correlation includes:
receiving an event for the partition;
determining whether the event is a partial match for the rule, wherein the partial match includes a partial satisfaction of the condition;
if the event is a partial match, determining whether the rule is a partition-independent rule or a partitioned rule;
if the rule is a partition-independent rule, sending information for the partial match to a rule designated node to aggregate events to determine whether the condition is satisfied; and
if the rule is a partitioned rule, storing the event at the cluster node as a partial match and aggregating events at the cluster node to determine whether the condition is satisfied.
16. The master node of claim 15 , wherein the at least one processor is to:
receive an event from a data source;
extract a value for the event field from the event;
determine whether the extracted value is a non-null value;
if the extracted value is non-null, identify a corresponding cluster node for the received event based on the block in which the extracted value is located; and
send the event to the corresponding cluster node.
17. The master node of claim 15 , wherein the at least one processor is to:
if the extracted value is null, send the event to an independent cluster node, wherein the independent cluster node is not assigned to any of the contiguous blocks of values for the event field and is to correlate received events according an event rule.
18. The master node of claim 15 , wherein the at least one processor is to:
determine whether a new cluster node is joining the cluster;
if the new cluster node is joining the cluster, assign a portion of the contiguous blocks of event field values to the new cluster node; and
update the partition map to include the new cluster node and its partition and its failover cluster node and to include any changes to the partitions as a result of the assigning of the portion of the contiguous blocks of event field values to the new cluster node.
19. The master node of claim 18 , wherein the new cluster node loads application data for its partition from an event correlation system data storage storing events for the distributed event correlation system.