Threat detection with event stream correlation using join keys

1 . A method comprising:

identifying a group of fields of a plurality of fields corresponding to a plurality of events having a percentage of cardinality and entropy of values greater than a defined threshold percentage level for the cardinality and the entropy as a group of join keys corresponding to the plurality of events that occurred within a defined timeframe;

determining which particular join keys of the group of join keys corresponding to the plurality of events that occurred within the defined timeframe are common join keys among the plurality of events;

applying a weight boosting function to at least one event of the plurality of events based on values in at least one field of the at least one event matching a predefined regular expression pattern stored in a database of regular expression patterns, wherein the weight boosting function applies a weight boosting value to the at least one event to increase influence of the at least one event on an event correlation confidence score;

correlating pairs of events in the plurality of events that occurred within the defined timeframe based on the common join keys between each pair of events to form a set of correlated event pairs for detecting threats;

generating the event correlation confidence score for each correlated event pair of the set of correlated event pairs based on the cardinality and the entropy of the values corresponding to each correlated event pair, number of common join keys used to correlate each correlated event pair, and the weight boosting value applied to any event in each correlated event pair

selecting those correlated event pairs of the set of correlated event pairs having the event correlation confidence score greater than an event correlation confidence score threshold level;

performing a threat detection analysis of the plurality of events that occurred within the defined timeframe utilizing only those correlated event pairs of the set of correlated event pairs having the event correlation confidence score greater than the event correlation confidence score threshold level to decrease threat detection and response time; and

determining whether a cybersecurity threat was detected based on the threat detection analysis of the plurality of events that occurred within the defined timeframe.

2 . The method of claim 1 , further comprising:

responsive to determining that the threat was detected based on the threat detection analysis of the plurality of events that occurred within the defined timeframe, performing a set of action steps to mitigate the threat.

3 . The method of claim 2 , wherein the set of action steps to mitigate the threat include at least one of sending a notification to a security analyst regarding the threat, isolating any affected devices, applying security patches to the affected devices, and reporting the threat to other threat detection systems.

4 . The method of claim 1 , further comprising:

receiving event logs from a plurality of different event log sources in real time;

performing an analysis of the event logs received from the plurality of different event log sources within the defined timeframe; and

identifying the plurality of events that occurred within the defined timeframe based on the analysis of the event logs received from the plurality of different event log sources.

5 . The method of claim 1 , further comprising:

identifying the plurality of fields corresponding to each of the plurality of events that occurred within the defined timeframe; and

performing an analysis of the values contained in each of the plurality of fields corresponding to each of the plurality of events that occurred within the defined timeframe.

6 . The method of claim 5 , further comprising:

determining the percentage of the cardinality and the entropy of the values based on the analysis of the values contained in each of the plurality of fields corresponding to each of the plurality of events that occurred within the defined timeframe; and

selecting the group of fields of the plurality of fields having the percentage of the cardinality and the entropy of the values greater than the defined threshold percentage level for the cardinality and the entropy.

7 . The method of claim 1 , wherein applying the weight boosting function comprises:

applying the weight boosting value to the at least one event at an event level to amplify a signal strength corresponding to the at least one event, wherein the database of regular expression patterns comprises a plurality of regular expression patterns organized for a specific type of values contained in the at least one field, and wherein the weight boosting value applied to the at least one event is determined based on a most specific regular expression pattern, of the plurality of regular expression patterns, that is matched by the values in the at least one field.

8 . The method of claim 1 , wherein the defined timeframe is dynamically defined such that a number of events of the plurality of events evaluated for correlation shifts depending on a predefined condition or rule corresponding to a threat detection use case, thereby enabling correlation of events occurring over an extended period of time.

9 . A computer system comprising:

a processor set;

one or more computer-readable storage media; and

program instructions stored on the one or more computer-readable storage media to cause the processor set to perform operations comprising:

identifying a group of fields of a plurality of fields corresponding to a plurality of events having a percentage of cardinality and entropy of values greater than a defined threshold percentage level for the cardinality and the entropy as a group of join keys corresponding to the plurality of events that occurred within a defined timeframe;

determining which particular join keys of the group of join keys corresponding to the plurality of events that occurred within the defined timeframe are common join keys among the plurality of events;

applying a weight boosting function to at least one event of the plurality of events based on values in at least one field of the at least one event matching a predefined regular expression pattern stored in a database of regular expression patterns, wherein the weight boosting function applies a weight boosting value to the at least one event to increase influence of the at least one event on an event correlation confidence score;

correlating pairs of events in the plurality of events that occurred within the defined timeframe based on the common join keys between each pair of events to form a set of correlated event pairs for detecting threats;

generating the event correlation confidence score for each correlated event pair of the set of correlated event pairs based on the cardinality and the entropy of the values corresponding to each correlated event pair, number of common join keys used to correlate each correlated event pair, and the weight boosting value applied to any event in each correlated event pair;

selecting those correlated event pairs of the set of correlated event pairs having the event correlation confidence score greater than an event correlation confidence score threshold level;

performing a threat detection analysis of the plurality of events that occurred within the defined timeframe utilizing only those correlated event pairs of the set of correlated event pairs having the event correlation confidence score greater than the event correlation confidence score threshold level to decrease threat detection and response time; and

determining whether a cybersecurity threat was detected based on the threat detection analysis of the plurality of events that occurred within the defined timeframe.

10 . The computer system of claim 9 , wherein the operations further comprise:

responsive to determining that the threat was detected based on the threat detection analysis of the plurality of events that occurred within the defined timeframe, performing a set of action steps to mitigate the threat.

11 . The computer system of claim 9 , wherein applying the weight boosting function comprises:

applying the weight boosting value to the at least one event at an event level to amplify a signal strength corresponding to the at least one event, wherein the database of regular expression patterns comprises a plurality of regular expression patterns organized for a specific type of values contained in the at least one field, and wherein the weight boosting value applied to the at least one event is determined based on a most specific regular expression pattern, of the plurality of regular expression patterns, that is matched by the values in the at least one field.

12 . The computer system of claim 9 , wherein the defined timeframe is dynamically defined such that a number of events of the plurality of events evaluated for correlation shifts depending on a predefined condition or rule corresponding to a threat detection use case, thereby enabling correlation of events occurring over an extended period of time.

13 . A computer program product comprising:

one or more computer-readable storage media; and

program instructions stored on the one or more computer-readable storage media to perform operations comprising:

identifying a group of fields of a plurality of fields corresponding to a plurality of events having a percentage of cardinality and entropy of values greater than a defined threshold percentage level for the cardinality and the entropy as a group of join keys corresponding to the plurality of events that occurred within a defined timeframe;

determining which particular join keys of the group of join keys corresponding to the plurality of events that occurred within the defined timeframe are common join keys among the plurality of events;

applying a weight boosting function to at least one event of the plurality of events based on values in at least one field of the at least one event matching a predefined regular expression pattern stored in a database of regular expression patterns, wherein the weight boosting function applies a weight boosting value to the at least one event to increase influence of the at least one event on an event correlation confidence score;

correlating pairs of events in the plurality of events that occurred within the defined timeframe based on the common join keys between each pair of events to form a set of correlated event pairs for detecting threats;

generating the event correlation confidence score for each correlated event pair of the set of correlated event pairs based on the cardinality and the entropy of the values corresponding to each correlated event pair, number of common join keys used to correlate each correlated event pair, and the weight boosting value applied to any event in each correlated event pair;

selecting those correlated event pairs of the set of correlated event pairs having the event correlation confidence score greater than an event correlation confidence score threshold level;

performing a threat detection analysis of the plurality of events that occurred within the defined timeframe utilizing only those correlated event pairs of the set of correlated event pairs having the event correlation confidence score greater than the event correlation confidence score threshold level to decrease threat detection and response time; and

determining whether a cybersecurity threat was detected based on the threat detection analysis of the plurality of events that occurred within the defined timeframe.

14 . The computer program product of claim 13 , wherein the operations further comprise:

responsive to determining that the threat was detected based on the threat detection analysis of the plurality of events that occurred within the defined timeframe, performing a set of action steps to mitigate the threat.

15 . The computer program product of claim 14 , wherein the set of action steps to mitigate the threat include at least one of sending a notification to a security analyst regarding the threat, isolating any affected devices, applying security patches to the affected devices, and reporting the threat to other threat detection systems.

16 . The computer program product of claim 13 , wherein the operations further comprise:

receiving event logs from a plurality of different event log sources in real time;

performing an analysis of the event logs received from the plurality of different event log sources within the defined timeframe; and

identifying the plurality of events that occurred within the defined timeframe based on the analysis of the event logs received from the plurality of different event log sources.

17 . The computer program product of claim 13 , wherein the operations further comprise:

identifying the plurality of fields corresponding to each of the plurality of events that occurred within the defined timeframe; and

performing an analysis of the values contained in each of the plurality of fields corresponding to each of the plurality of events that occurred within the defined timeframe.

18 . The computer program product of claim 17 , wherein the operations further comprise:

determining the percentage of the cardinality and the entropy of the values based on the analysis of the values contained in each of the plurality of fields corresponding to each of the plurality of events that occurred within the defined timeframe; and

selecting the group of fields of the plurality of fields having the percentage of the cardinality and the entropy of the values greater than the defined threshold percentage level for the cardinality and the entropy.

19 . The computer program product of claim 13 , wherein applying the weight boosting function comprises:

applying the weight boosting value to the at least one event at an event level to amplify a signal strength corresponding to the at least one event, wherein the database of regular expression patterns comprises a plurality of regular expression patterns organized for a specific type of values contained in the at least one field, and wherein the weight boosting value applied to the at least one event is determined based on a most specific regular expression pattern, of the plurality of regular expression patterns, that is matched by the values in the at least one field.

20 . The computer program product of claim 13 , wherein the defined timeframe is dynamically defined such that a number of events of the plurality of events evaluated for correlation shifts depending on a predefined condition or rule corresponding to a threat detection use case, thereby enabling correlation of events occurring over an extended period of time.