System, method, and computer program for context enrichment of logs in a cybersecurity system
The present disclosure describes a system, method, and computer program for enriching logs with cybersecurity threat information. The system receives a first input data stream with logs usable for cybersecurity evaluation and a second input data stream with data values (e.g., IP address, user IDs, etc.) that are associated with a cybersecurity threat (“threat values”). The system stores the threat values in a datastore as they are received. As logs are received, the system enriches each log having a data value that matches a threat value in the datastore. As compared to conventional systems, the system significantly reduces the number of context queries on the datastore by using a probabilistic filter to determine if there is a zero or non-zero probability of a data value in a log being one of the threat values. If there is a zero probability, the system is able to determine that the log does not have one of the threat values and, thus, can enrich the log based solely on the output of the probabilistic filter.
1 . A non-transitory computer-readable medium comprising a computer program, that, when executed by a computer system, enables the computer system to perform the following method for enriching logs with cybersecurity threat information, the method comprising:
receiving a first input data stream and a second input data stream, wherein the first input data stream includes logs usable for cybersecurity evaluation and the second input data stream includes data values that are associated with a cybersecurity threat;
storing the data values associated with the cybersecurity threat in a first datastore;
for each log in the first input data stream having a select data field, processing the log by performing the following:
identifying a data value in the select data field;
using a probabilistic filter to determine whether there is a zero probability of the first datastore having a data value matching the data value in the select data field;
in response to there being a zero probability of the first datastore having the matching data value, determining that there is no indication in the first datastore that the data value in the select data field is associated with a cybersecurity threat;
in response to there being a non-zero probability of the first datastore having the matching data value, searching the first datastore for the matching data value and, in response to finding the matching data value in the first datastore, enriching the log to indicate that the data value in the select data field is associated with a cybersecurity threat; and
outputting the log into an output data stream.
2 . The non-transitory computer-readable medium of claim 1 , further comprising:
in response to there being a zero probability of the first datastore having the matching data value, enriching the log to indicate that the data value in the select data field is not associated with a cybersecurity threat.
3 . The non-transitory computer-readable medium of claim 1 , wherein the logs relate to one or more computer-based systems being monitored by a cybersecurity system.
4 . The non-transitory computer-readable medium of claim 1 , wherein the data values in the first datastore are IP addresses associated with one or more of the following: ransomware, phishing, malware, and a trojan attack.
5 . The non-transitory computer-readable medium of claim 1 , wherein the data values in the first datastore are user IDs associated with one or more of the following: a compromised user, a malicious user, a watched user, and a high value user.
6 . The non-transitory computer-readable medium of claim 1 , wherein a subset of the data values in the second input data stream are stored in a second datastore and wherein prior to the step of using the probabilistic filter, the method further comprises:
for each log having the select data field, searching the second datastore for a data value matching the data value in the select data field;
in response to finding the matching data value in the second datastore, enriching the log to indicate that the data value in the select data field is associated with a cybersecurity threat, bypassing the probabilistic filter and the first datastore, and outputting the enriched log in the output data stream; and
in response to not finding the matching data value in the second datastore, proceeding with the step of using the probabilistic filter.
7 . The non-transitory computer-readable medium of claim 6 , wherein the first datastore is a distributed datastore and the second datastore is a local datastore.
8 . The non-transitory computer-readable medium of claim 7 , wherein the first datastore is a distributed in-memory cache and the second datastore is a local cache running on the system on which the logs are received.
9 . The non-transitory computer-readable medium of claim 7 , wherein, in response to searching and finding the matching data value in the distributed in-memory cache, adding the matching data value to the local cache.
10 . A computer system for enriching logs with cybersecurity threat information, the system comprising:
one or more processors;
one or more memory units coupled to the one or more processors, wherein the one or more memory units store instructions that, when executed by the one or more processors, cause the system to perform the operations of:
receiving a first input data stream and a second input data stream, wherein the first input data stream includes logs usable for cybersecurity evaluation and the second input data stream includes data values that are associated with a cybersecurity threat;
storing the data values associated with the cybersecurity threat in a first datastore;
for each log in the first input data stream having a select data field, processing the log by performing the following:
identifying a data value in the select data field;
using a probabilistic filter to determine whether there is a zero probability of the first datastore having a data value matching the data value in the select data field;
in response to there being a zero probability of the first datastore having the matching data value, determining that there is no indication in the first datastore that the data value in the select data field is associated with a cybersecurity threat;
in response to there being a non-zero probability of the first datastore having the matching data value, searching the first datastore for the matching data value and, in response to finding the matching data value in the first datastore, enriching the log to indicate that the data value in the select data field is associated with a cybersecurity threat; and
outputting the log into an output data stream.
11 . The system of claim 10 , further comprising:
in response to there being a zero probability of the first datastore having the matching data value, enriching the log to indicate that the data value in the select data field is not associated with a cybersecurity threat.
12 . The system of claim 10 , wherein the data values in the first datastore are IP addresses associated with one or more of the following: ransomware, phishing, malware, and a trojan attack.
13 . The system of claim 10 , wherein the data values in the first datastore are user IDs associated with one or more of the following: a compromised user, a malicious user, a watched user, and a high value user.
14 . The system of claim 10 , wherein a subset of the data values in the second input data stream are stored in a second datastore and wherein prior to the step of using the probabilistic filter, the method further comprises:
for each log having the select data field, searching the second datastore for a data value matching the data value in the select data field;
in response to finding the matching data value in the second datastore, enriching the log to indicate that the data value in the select data field is associated with a cybersecurity threat, bypassing the probabilistic filter and the first datastore, and outputting the enriched log in the output data stream; and
in response to not finding the matching data value in the second datastore, proceeding with the step of using the probabilistic filter.
15 . The system of claim 14 , wherein the first datastore is a distributed in-memory cache and the second datastore is a local cache running on the computer on which the logs are received.
16 . A method, performed by a computer-based cybersecurity system, for enriching logs with cybersecurity threat information, the method comprising:
receiving a first input data stream and a second input data stream, wherein the first input data stream includes logs usable for cybersecurity evaluation and the second input data stream includes data values that are associated with a cybersecurity threat;
storing the data values associated with the cybersecurity threat in a first datastore;
for each log in the first input data stream having a select data field, processing the log by performing the following:
identifying a data value in the select data field;
using a probabilistic filter to determine whether there is a zero probability of the first datastore having a data value matching the data value in the select data field;
in response to there being a zero probability of the first datastore having the matching data value, determining that there is no indication in the first datastore that the data value in the select data field is associated with a cybersecurity threat;
in response to there being a non-zero probability of the first datastore having the matching data value, searching the first datastore for the matching data value and, in response to finding the matching data value in the first datastore, enriching the log to indicate that the data value in the select data field is associated with a cybersecurity threat; and
outputting the enriched log into an output data stream.
17 . The method of claim 16 further comprising:
in response to there being a zero probability of the first datastore having the matching data value, enriching the log to indicate that the data value in the select data field is not associated with a cybersecurity threat.
18 . The method of claim 16 , wherein the data values in the first datastore are IP addresses associated with one or more of the following: ransomware, phishing, malware, and a trojan attack.
19 . The method of claim 16 , wherein the data values in the first datastore are user IDs associated with one or more of the following: a compromised user, a malicious user, a watched user, and a high value user.
20 . The method of claim 16 , wherein a subset of the data values in the second input data stream are stored in a second datastore and wherein prior to the step of using the probabilistic filter, the method further comprises:
for each log having the select data field, searching the second datastore for a data value matching the data value in the select data field;
in response to finding the matching data value in the second datastore, enriching the log to indicate that the data value in the select data field is associated with a cybersecurity threat, bypassing the probabilistic filter and the first datastore, and outputting the enriched log in the output data stream; and
in response to not finding the matching data value in the second datastore, proceeding with the step of using the probabilistic filter.