Distributed alert and suppression management in a cluster computing system
A first processing node of a cluster of processing nodes issues a first alert when first event data satisfies a trigger condition, and sends, to an alert data store external to the cluster, a first alert record of the first alert and suppression information based at least in part on the first alert. A second processing node of the cluster determines that second event data satisfies the trigger condition, obtains, from the alert data store, the suppression information indicating that an expiration time for suppressing the first alert is unexpired, and sends, to the alert data store, a second alert record of a second alert without issuing the second alert.
1. A computer-implemented method, comprising:
executing, by a first processing node of a cluster of processing nodes, a query to identify when first event data observed by the first processing node satisfies a trigger condition;
issuing, by the first processing node, a first alert when the first event data, identified by the first processing node responsive to executing the query, satisfies the trigger condition;
sending, by the first processing node and to a central alert data store external to the cluster and the first processing node, a first alert record of the first alert and a suppression record comprising a suppression key uniquely identifying the suppression record and a suppression identifier identifying the query, wherein the sending the first alert record and the suppression record is based at least in part on the first alert;
executing, by a second processing node of the cluster of processing nodes, the query to identify when second event data observed by the second processing node satisfies the trigger condition, wherein the second processing node has dedicated processing resources and is separate from the first processing node and the central alert data store;
determining, by the second processing node of the cluster, that the second event data, identified by the second processing node responsive to the query, satisfies the trigger condition;
obtaining, by the second processing node and from the central alert data store using the suppression identifier identifying the query, the suppression record indicating that an expiration time for suppressing the first alert is unexpired; and
sending, by the second processing node and to the central alert data store, a second alert record of a second alert without issuing the second alert.
2. The computer-implemented method of claim 1 , further comprising:
failing, by the first processing node, to obtain from the central alert data store, the suppression record for the first alert, wherein the first processing node issues the first alert record in response to the failing.
3. The computer-implemented method of claim 1 , wherein the first event data comprises results of executing a search query.
4. The computer-implemented method of claim 1 , further comprising:
after sending the second alert record to the central alert data store, determining, by the second processing node, that the expiration time for suppressing the first alert has expired; and
issuing, by the second processing node, the second alert.
5. The computer-implemented method of claim 1 , further comprising:
after sending the second alert record to the central alert data store without issuing the second alert, issuing, by the second processing node and based on the first alert specifying a digest mode, a third alert when third event data satisfies the trigger condition.
6. The computer-implemented method of claim 1 , wherein the suppression record specifies one or more fields, wherein the first event data comprises first values of the one or more fields, and wherein the method further comprises:
recording, in a list of recorded values and for a series of events, unique combinations of values of the one or more fields; and
determining that the first values are included in the list of recorded values, wherein sending the second alert record without issuing the second alert is based on the determining.
7. The computer-implemented method of claim 1 , further comprising:
receiving, by the second processing node, a request for a graphical user interface (GUI);
obtaining, by the second processing node and from the central alert data store, a plurality of alert records; and
updating the GUI using the plurality of alert records.
8. The computer-implemented method of claim 1 , further comprising:
deleting, by the central alert data store, the suppression record when a current time exceeds an expiration time of the suppression record.
9. The computer-implemented method of claim 1 , wherein the suppression record specifies a group of alerts including the first alert, the method further comprising:
setting an expiration time of the suppression record to an expiration time of the first alert.
10. The computer-implemented method of claim 1 , wherein the suppression record specifies a group of alerts including the first alert, the method further comprising:
setting an expiration time of the suppression record to an expiration time of the first alert to obtain an updated suppression record; and
sending, by the second processing node and to the central alert data store, the updated suppression record.
11. The computer-implemented method of claim 1 , wherein the suppression record specifies a group of alerts including the first alert and a third alert, the method further comprising:
setting an expiration time of the suppression record to a first expiration time of the first alert,
wherein the third alert specifies a second expiration time different from the first expiration time.
12. The computer-implemented method of claim 1 , further comprising:
sending, by the first processing node and to the central alert data store, a third alert record of a third alert; and
restricting, to the first processing node and by the central alert data store, access to the third alert record.
13. The computer-implemented method of claim 1 , the central alert data store stores a plurality of suppression records corresponding to a plurality of names of search queries, the method further comprising:
obtaining, from the central alert data store, a subset of the plurality of suppression information corresponding to a subset of the plurality of names of search queries.
14. A computing device, comprising:
a processor; and
a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations including:
executing, by a first processing node of a cluster of processing nodes, a query to identify when first event data observed by the first processing node satisfies a trigger condition,
issuing, by the first processing node, a first alert when the first event data, identified by the first processing node responsive to executing the query, satisfies the trigger condition,
sending, by the first processing node and to a central alert data store external to the cluster and the first processing node, a first alert record of the first alert and a suppression record comprising a suppression key uniquely identifying the suppression record and a suppression identifier identifying the query, wherein the sending the first alert record and the suppression record is based at least in part on the first alert,
executing, by a second processing node of the cluster of processing nodes, the query to identify when second event data observed by the second processing node satisfies the trigger condition, wherein the second processing node has dedicated processing resources and is separate from the first processing node and the central alert data store,
determining, by the second processing node of the cluster, that the second event data, identified by the second processing node responsive to the query, satisfies the trigger condition,
obtaining, by the second processing node and from the central alert data store using the suppression identifier identifying the query, the suppression record indicating that an expiration time for suppressing the first alert is unexpired, and
sending, by the second processing node and to the central alert data store, a second alert record of a second alert without issuing the second alert.
15. The computing device of claim 14 , wherein the operations further comprise:
failing, by the first processing node, to obtain from the central alert data store, the suppression record for the first alert, wherein the first processing node issues the first alert record in response to the failing.
16. The computing device of claim 14 , wherein the operations further comprise:
after sending the second alert record to the central alert data store, determining, by the second processing node, that the expiration time for suppressing the first alert has expired; and
issuing, by the second processing node, the second alert.
17. The computing device of claim 14 , wherein the operations further comprise:
after sending the second alert record to the central alert data store without issuing the second alert, issuing, by the second processing node and based on the first alert specifying a digest mode, a third alert when third event data satisfies the trigger condition.
18. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processor to perform operations including:
executing, by a first processing node of a cluster of processing nodes, a query to identify when first event data observed by the first processing node satisfies a trigger condition;
issuing, by the first processing node, a first alert when the first event data, identified by the first processing node responsive to executing the query, satisfies the trigger condition;
sending, by the first processing node and to a central alert data store external to the cluster and the first processing node, a first alert record of the first alert and a suppression record comprising a suppression key uniquely identifying the suppression record and a suppression identifier identifying the query, wherein the sending the first alert record and the suppression record is based at least in part on the first alert;
executing, by a second processing node of the cluster of processing nodes, the query to identify when second event data observed by the second processing node satisfies the trigger condition, wherein the second processing node has dedicated processing resources and is separate from the first processing node and the central alert data store;
determining, by the second processing node of the cluster, that the second event data, identified by the second processing node responsive to the query, satisfies the trigger condition;
obtaining, by the second processing node and from the central alert data store using the suppression identifier identifying the query, the suppression record indicating that an expiration time for suppressing the first alert is unexpired; and
sending, by the second processing node and to the central alert data store, a second alert record of a second alert without issuing the second alert.
19. The non-transitory computer-readable medium of claim 18 , wherein the operations further comprise:
failing, by the first processing node, to obtain from the central alert data store, the suppression record for the first alert, wherein the first processing node issues the first alert record in response to the failing.
20. The non-transitory computer-readable medium of claim 18 , wherein the operations further comprise:
after sending the second alert record to the central alert data store without issuing the second alert, issuing, by the second processing node and based on the first alert specifying a digest mode, a third alert when third event data satisfies the trigger condition.