Event correlation determination in extended detection and response systems
This disclosure describes techniques for evaluating a correlation between two monitoring events based on a regularized co-occurrence occurrence measure associated with the two monitoring events. For example, in some cases, the techniques described herein include determining a co-occurrence measure associated with two monitoring events by regularizing an initial co-occurrence measure based on the respective occurrence measures associated with the two monitoring events. In some cases, an example system: (i) determines a first occurrence measure associated with a first event and a second occurrence measure associated with a second event, (ii) determines a co-occurrence measure associated with the two events, (iii) determines a regularization parameter based on the first and second occurrence measures as well as the co-occurrence measure, and (iv) determines a regularized co-occurrence measure based on the co-occurrence measure and the regularization parameter.
1 . A method comprising:
receiving, by a processor, a first monitoring event associated with a security incident recorded in relation to a computer system, wherein the first monitoring event is associated with a first event category and a first feature value corresponding to a first feature type;
receiving, by the processor, a second monitoring event associated with the computer system, wherein the second monitoring event is associated with a second event category and a second feature value corresponding to a second feature type;
determining first data representing that the first event category is indexed based at least in part on the first feature type on an event repository;
determining second data representing that the second event category is indexed based at least in part on the second feature type on the event repository;
determining, by the processor and based on the first data, a first identifier associated with the first monitoring event based on the first feature value;
determining, by the processor and based on the second data, a second identifier associated with the second monitoring event based on the second feature value;
querying, by the processor, the event repository based on the first identifier and a third identifier associated with the first event category to determine a first query output;
determining, by the processor and based on the first query output, a first occurrence measure associated with the first monitoring event;
querying, by the processor, the event repository based on the second identifier and a fourth identifier associated with the second event category to determine a second query output;
determining, by the processor and based on the second query output, a second occurrence measure associated with the second monitoring event;
determining, by the processor, a co-occurrence measure associated with the first monitoring event and the second monitoring event;
determining, by the processor, a ranking score associated with the second monitoring event based on the first occurrence measure, the second occurrence measure, and the co-occurrence measure, wherein determining the co-occurrence measure comprises:
determining, by querying the event repository based on the first identifier, a first set of occurrences of the first monitoring event,
determining, by querying the event repository based on the second identifier, a second set of occurrences of the second monitoring event,
determining, based on a first timestamp associated with a first occurrence in the first set of occurrences and a second timestamp associated with a second occurrence in the second set of occurrences, first data representing that the second occurrence has occurred within a first threshold period of the first occurrence,
determining, based on a third timestamp associated with a third occurrence in the first set of occurrences and a fourth timestamp associated with a fourth occurrence in the second set of occurrences, second data representing that the third occurrence has occurred within a second threshold period of the fourth occurrence, and
determining the co-occurrence measure based on the first data and the second data; and
based on determining that the ranking score satisfies a threshold, causing, by the processor, a representation of the second monitoring event to be displayed using a system administrator platform associated with the computer system.
2 . The method of claim 1 , wherein the first event category is associated with a first event monitoring component and the second event category is associated with a second event monitoring component.
3 . The method of claim 1 , wherein determining the ranking score comprises:
determining a regularization parameter based on the first occurrence measure and the second occurrence measure, wherein the regularization parameter represents a combined occurrence frequency associated with the first monitoring event and the second monitoring event; and
determining the ranking score based on a transformation of the co-occurrence measure based on the regularization parameter.
4 . The method of claim 3 , wherein the regularization parameter is determined based on the first occurrence measure, the second occurrence measure, and the co-occurrence measure.
5 . The method of claim 3 , wherein determining the regularization parameter comprises:
querying the event repository based on the third identifier to determine a third occurrence measure associated with the first event category;
querying the event repository based on the fourth identifier to determine a fourth occurrence measure associated with the second event category; and
determining the regularization parameter based on the first occurrence measure, the second occurrence measure, the third occurrence measure, and the fourth occurrence measure.
6 . The method of claim 1 , wherein determining the ranking score comprises:
determining the ranking score based on a machine learning model, wherein input data to the machine learning model comprises the first occurrence measure, the second occurrence measure, and the co-occurrence measure.
7 . The method of claim 6 , further comprising:
receiving, by the processor, an indication of removal of the second monitoring event from a list displayed using the system administrator platform; and
based on receiving the indication, retraining the machine learning model.
8 . The method of claim 1 , wherein the first monitoring event is selected by a user of the system administrator platform.
9 . The method of claim 1 , wherein:
events associated with the first event category are indexed based on a first set of feature types, and
events associated with the second event category are indexed based on a second set of feature types, wherein the first set of feature types are different from the second set of feature types.
10 . A system comprising:
one or more processors; and
one or more computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving a first monitoring event associated with a security incident recorded in relation to a computer system, wherein the first monitoring event is associated with a first event category and a first feature value corresponding to a first feature type;
receiving a second monitoring event associated with the computer system, wherein the second monitoring event is associated with a second event category and a second feature value corresponding to a second feature type;
determining first data representing that the first event category is indexed based at least in part on the first feature type on an event repository;
determining second data representing that the second event category is indexed based at least in part on the second feature type on the event repository;
determining, based on the first data, a first identifier associated with the first monitoring event based on the first feature value;
determining, based on the second data, a second identifier associated with the second monitoring event based on the second feature value;
querying the event repository based on the first identifier and a third identifier associated with the first event category to determine a first query output;
determining, based on the first query output, a first occurrence measure associated with the first monitoring event;
querying the event repository based on the second identifier and a fourth identifier associated with the second event category to determine a second query output;
determining, based on the second query output, a second occurrence measure associated with the second monitoring event;
determining a co-occurrence measure associated with the first monitoring event and the second monitoring event, wherein determining the co-occurrence measure comprises:
determining, by querying the event repository based on the first identifier, a first set of occurrences of the first monitoring event,
determining, by querying the event repository based on the second identifier, a second set of occurrences of the second monitoring event,
determining, based on a first timestamp associated with a first occurrence in the first set of occurrences and a second timestamp associated with a second occurrence in the second set of occurrences, first data representing that the second occurrence has occurred within a first threshold period of the first occurrence,
determining, based on a third timestamp associated with a third occurrence in the first set of occurrences and a fourth timestamp associated with a fourth occurrence in the second set of occurrences, second data representing that the third occurrence has occurred within a second threshold period of the fourth occurrence, and
determining the co-occurrence measure based on the first data and the second data;
determining a ranking score associated with the second monitoring event based on the first occurrence measure, the second occurrence measure, and the co-occurrence measure; and
based on determining that the ranking score satisfies a threshold, causing a representation of the second monitoring event to be displayed using a system administrator platform associated with the computer system.
11 . The system of claim 10 , wherein determining the ranking score comprises:
determining a regularization parameter based on the first occurrence measure and the second occurrence measure, wherein the regularization parameter represents a combined occurrence frequency associated with the first monitoring event and the second monitoring event; and
determining the ranking score based on a transformation of the co-occurrence measure based on the regularization parameter.
12 . The system of claim 11 , wherein the regularization parameter is determined based on the first occurrence measure, the second occurrence measure, and the co-occurrence measure.
13 . The system of claim 11 , wherein determining the regularization parameter comprises:
querying the event repository based on the third identifier to determine a third occurrence measure associated with the first event category;
querying the event repository based on the fourth identifier to determine a fourth occurrence measure associated with the second event category; and
determining the regularization parameter based on the first occurrence measure, the second occurrence measure, the third occurrence measure, and the fourth occurrence measure.
14 . The system of claim 10 , wherein determining the ranking score comprises:
determining the ranking score based on a machine learning model, wherein input data to the machine learning model comprises the first occurrence measure, the second occurrence measure, and the co-occurrence measure.
15 . The system of claim 14 , the operations further comprising:
receiving an indication of removal of the second monitoring event from a list displayed using the system administrator platform; and
based on receiving the indication, retraining the machine learning model.
16 . The system of claim 10 , wherein the first monitoring event is selected by a user of the system administrator platform.
17 . One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving a first monitoring event associated with a security incident recorded in relation to a computer system, wherein the first monitoring event is associated with a first event category and a first feature value corresponding to a first feature type;
receiving a second monitoring event associated with the computer system, wherein the second monitoring event is associated with a second event category and a second feature value corresponding to a second feature type;
determining first data representing that the first event category is indexed based at least in part on the first feature type on an event repository;
determining second data representing that the second event category is indexed based at least in part on the second feature type on the event repository;
determining, based on the first data, a first identifier associated with the first monitoring event based on the first feature value;
determining, based on the second data, a second identifier associated with the second monitoring event based on the second feature value;
querying the event repository based on the first identifier and a third identifier associated with the first event category to determine a first query output;
determining, based on the first query output, a first occurrence measure associated with the first monitoring event;
querying the event repository based on the second identifier and a fourth identifier associated with the second event category to determine a second query output;
determining, based on the second query output, a second occurrence measure associated with the second monitoring event;
determining a co-occurrence measure associated with the first monitoring event and the second monitoring event, wherein determining the co-occurrence measure comprises:
determining, by querying the event repository based on the first identifier, a first set of occurrences of the first monitoring event,
determining, by querying the event repository based on the second identifier, a second set of occurrences of the second monitoring event,
determining, based on a first timestamp associated with a first occurrence in the first set of occurrences and a second timestamp associated with a second occurrence in the second set of occurrences, first data representing that the second occurrence has occurred within a first threshold period of the first occurrence,
determining, based on a third timestamp associated with a third occurrence in the first set of occurrences and a fourth timestamp associated with a fourth occurrence in the second set of occurrences, second data representing that the third occurrence has occurred within a second threshold period of the fourth occurrence, and
determining the co-occurrence measure based on the first data and the second data;
determining a ranking score associated with the second monitoring event based on the first occurrence measure, the second occurrence measure, and the co-occurrence measure; and
based on determining that the ranking score satisfies a threshold, causing a representation of the second monitoring event to be displayed using a system administrator platform associated with the computer system.
18 . The one or more non-transitory computer-readable media of claim 17 , wherein determining the ranking score comprises:
determining a regularization parameter based on the first occurrence measure and the second occurrence measure, wherein the regularization parameter represents a combined occurrence frequency associated with the first monitoring event and the second monitoring event; and
determining the ranking score based on a transformation of the co-occurrence measure based on the regularization parameter.
19 . The one or more non-transitory computer-readable media of claim 18 , wherein the regularization parameter is determined based on the first occurrence measure, the second occurrence measure, and the co-occurrence measure.