Streaming and filtering event objects into a data lake
An asynchronous stream of security events is added to a data lake for enterprise security by identifying groups of related events related to a security threat, and creating rules to fold these related events into a single security event along with metadata. The folding rules may then be applied to security events in the event stream to compress data in the data lake and improve detection efficiency.
1 . A computer program product comprising computer executable code embodied in a non-transitory computer readable medium that, when executing on one or more computing devices, causes the one or more computing devices to perform the steps of:
storing a data lake containing a plurality of security events received on an event stream in an enterprise network;
detecting a behavior of interest in the enterprise network at a threat management facility based on the plurality of security events stored in the data lake;
identifying a characteristic sequence of security events in the event stream preceding the behavior of interest;
creating a folding rule for combining the characteristic sequence of security events into an aggregated security event in the data lake, and for augmenting the aggregated security event with descriptive metadata;
transmitting the folding rule to a local security agent executing on a compute instance in the enterprise network for use in monitoring local activity on the compute instance;
receiving an instance of the aggregated security event at the threat management facility from the compute instance; and
storing the instance of the aggregated security event in the data lake.
2 . The computer program product of claim 1 , wherein the event stream includes one or more events received from data recorders on a plurality of compute instances associated with the enterprise network.
3 . The computer program product of claim 1 , wherein the plurality of security events includes an asynchronous stream of security events from the enterprise network, the asynchronous stream including a combination of batch transfers including groups of security events and streaming transfers of individual security events from one or more compute instances in the enterprise network.
4 . The computer program product of claim 1 , further comprising code that causes the one or more computing devices to perform the step of receiving a manual approval of the folding rule at an administrative console for the threat management facility before transmitting the folding rule to the compute instance.
5 . The computer program product of claim 1 , wherein the characteristic sequence of security events includes a sequence of similar events from a plurality of compute instances in the enterprise network or a sequence of similar events from a single compute instance in the enterprise network.
6 . The computer program product of claim 1 , wherein the descriptive metadata characterizes a number of security events in the characteristic sequence of security events.
7 . The computer program product of claim 1 , wherein the descriptive metadata characterizes a frequency of events in the characteristic sequence of security events.
8 . The computer program product of claim 1 , wherein the descriptive metadata characterizes a first and last security event in the characteristic sequence of security events.
9 . The computer program product of claim 1 , further comprising code that causes the one or more computing devices to perform the step of performing a malware detection for the enterprise network with the threat management facility using the instance of the aggregated security event stored in the data lake.
10 . The computer program product of claim 1 , further comprising code that causes the one or more computing devices to perform the step of initiating a remediation of a threat to the enterprise network with the threat management facility based on one or more instances of the aggregated security event stored in the data lake.
11 . The computer program product of claim 1 , further comprising code that causes the one or more computing devices to perform the step of filtering each of the plurality of security events in the event stream with a deduplication lookup before adding to the data lake.
12 . The computer program product of claim 1 , further comprising code that causes the one or more computing devices to perform the step of storing the plurality of security events in the data lake as one or more data objects each augmented with a description organized according to a schema.
13 . The computer program product of claim 1 , wherein detecting the behavior of interest includes applying a query to the data lake from an administrative console of the threat management facility.
14 . A computer program product comprising computer executable code embodied in a non-transitory computer readable medium that, when executing on one or more computing devices, causes the one or more computing devices to perform the steps of:
detecting a plurality of security events with a local security agent executing on a compute instance in an enterprise network;
transmitting the plurality of security events to an event stream for storage in a data lake of a threat management facility for the enterprise network;
receiving a folding rule at the local security agent from the threat management facility, the folding rule based on the plurality of security events, and the folding rule executable by the local security agent to combine a characteristic sequence of security events in the plurality of security events into an aggregated security event, and to augment the aggregated security event with descriptive metadata;
applying the folding rule on the compute instance to create an instance of the aggregated security event; and
transmitting the instance of the aggregated security event to the threat management facility for the storage in the data lake.
15 . The computer program product of claim 14 , wherein the plurality of security events includes an asynchronous stream of security events from the enterprise network, the asynchronous stream including a combination of batch transfers including groups of security events and streaming transfers of individual security events from one or more compute instances in the enterprise network.
16 . The computer program product of claim 14 , wherein the characteristic sequence of security events includes a sequence of similar events from a plurality of compute instances in the enterprise network or a sequence of similar events from a single compute instance in the enterprise network.
17 . The computer program product of claim 14 , wherein the descriptive metadata characterizes a number of security events in the characteristic sequence of security events.
18 . The computer program product of claim 14 , wherein the descriptive metadata characterizes a frequency of events in the characteristic sequence of security events.
19 . The computer program product of claim 14 , wherein the descriptive metadata characterizes a first and last security event in the characteristic sequence of security events.
20 . A system comprising:
a plurality of compute instances associated with an enterprise network;
a data lake including a data storage medium storing a first plurality of data objects representing security events within the enterprise network and a plurality of descriptions for the first plurality of data objects, each of the plurality of descriptions organized according to one or more schemas;
a stream service implemented by at least one processor to receive an asynchronous event stream of additional data objects representing the security events from the plurality of compute instances associated with the enterprise network;
a transformer service implemented by the at least one processor, the transformer service configured to:
process the asynchronous event stream by filtering the additional data objects to remove duplicate one or more data objects already stored in the data lake, thereby providing filtered data objects,
augment each of the filtered data objects with a corresponding description organized according to one of the one or more schemas, thereby providing augmented data objects,
store the augmented data objects in the data lake, and
create a folding rule for aggregating a sequence of similar events into an aggregated security event for storage in the data lake, and for augmenting the aggregated security event with metadata that characterizes one or more of a number of events in the sequence of similar events; and
a threat management facility configured to transmit the folding rule to the plurality of compute instances for use in monitoring activity in the enterprise network.