Multi-Stage Network Security Threat Detection

1 . A method comprising:

inputting, to an anomaly decision engine, feature sets of event data originating from a plurality of data sources on a computer network;

detecting, by the anomaly decision engine, network security anomalies based on the feature sets;

outputting, to a threat decision engine, anomaly data indicative of the detected network security anomalies;

detecting, by the threat decision engine, a network security threat based on the anomaly data; and

outputting, via a user interface, threat data indicative of the detected network security threat.

2 . A method as recited in claim 1 , further comprising, prior to inputting the feature sets to the anomaly decision engine:

receiving the event data in raw form; and

processing the event data in raw form through an extract-transform-load (ETL) process to produce the feature sets.

3 . A method as recited in claim 1 , further comprising, prior to inputting the feature sets to the anomaly decision engine:

receiving the event data in raw form; and

processing the event data in raw form through an extract-transform-load (ETL) process to produce the feature sets;

wherein the detecting of the anomalies is performed in real time as the event data are received by the ETL process.

4 . A method as recited in claim 1 , further comprising performing at least one of:

continuously training the anomaly decision engine based on the feature sets of event data, concurrently with said detecting security anomalies; or

continuously training the threat decision engine based on the anomaly data, concurrently with detecting security threats.

5 . A method as recited in claim 1 , wherein the event data comprises machine data.

6 . A method as recited in claim 1 , wherein the event data comprises timestamped machine data.

7 . A method as recited in claim 1 , further comprising:

outputting at least a portion of the anomaly data via the user interface.

8 . A method as recited in claim 1 , wherein the anomaly decision engine comprises a plurality of anomaly models and the threat decision engine comprises a plurality of threat models.

9 . A method as recited in claim 1 , wherein the anomaly decision engine comprises a plurality of machine learning anomaly models, each including logic to detect a different type of anomaly.

10 . A method as recited in claim 1 , wherein the threat decision engine comprises a plurality of machine learning threat models, each including logic to detect a different type of threat.

11 . A method as recited in claim 1 , wherein:

the anomaly decision engine comprises a plurality of machine learning anomaly models, each including logic to detect a different type of anomaly; and

the threat decision engine comprises a plurality of machine learning threat models, each including logic to detect a different type of threat.

12 . A method as recited in claim 1 , wherein the detecting of the anomalies and the detecting of the threat are performed in real time as the event data are received from the plurality of data sources.

13 . A method as recited in claim 1 , wherein the feature sets of event data are stored in a persistent storage system, and wherein the detecting of the anomalies and the detecting of the threat are performed in a batch processing mode based on the stored feature sets.

14 . A method as recited in claim 1 , wherein the detecting of the anomalies and the detecting of the threat are performed in real time as the event data are received from the plurality of data sources;

the method further comprising, in a batch processing mode:

detecting additional network security anomalies based on stored feature sets of event data originating from the plurality of data sources on the computer network; and

detecting an additional network security threat based on the detected additional network security anomalies.

15 . A method as recited in claim 1 , wherein:

the detecting of the anomalies and the detecting of the threat are performed in real time as the event data are received from the plurality of data sources;

the detecting of the anomalies includes using a first plurality of instances of a first plurality of machine learning models;

the detecting of the threat includes using a first plurality of instances of a second plurality of machine learning models;

the method further comprising, in a batch processing mode:

detecting additional network security anomalies, based on stored feature sets of event data originating from the plurality of data sources on the computer network, by using a second plurality of instances of the first plurality of machine learning models; and

detecting an additional network security threat based on the detected additional network security anomalies, by using a second plurality of instances of the second plurality of machine learning models.

16 . A method as recited in claim 1 , wherein the detecting of the anomalies and the detecting of the threat are performed in real time as the event data are received from the plurality of data sources;

the method further comprising:

in a batch processing mode,

inputting, to a batch anomaly decision engine, stored feature sets of event data originating from a plurality of data sources on a computer network;

detecting, by the batch anomaly decision engine, additional network security anomalies based on the stored feature sets;

outputting, to a batch threat decision engine, batch anomaly data indicative of the additional detected network security anomalies; and

detecting, by the batch threat decision engine, an additional network security threat based on the batch anomaly data.

17 . A method as recited in claim 1 , wherein the detecting of the anomalies and the detecting of the threat are performed in real time by a task-parallel distributed computing engine as the event data are received from the plurality of data sources;

the method further comprising:

in a batch processing mode,

inputting stored feature sets of event data to a batch anomaly decision engine implemented by a data parallel distributed computing engine, the stored feature sets being stored in a distributed file system, the event data associated with the feature sets originating from the plurality of data sources on the computer network;

detecting, by the batch anomaly decision engine, additional network security anomalies based on the stored feature sets;

outputting, to a batch threat decision engine, batch anomaly data indicative of the additional detected network security anomalies;

detecting, by the batch threat decision engine, an additional network security threat based on the batch anomaly data.

18 . A method as recited in claim 1 , wherein the detecting of the anomalies and the detecting of the threat are performed by using a task-parallel distributed processing engine.

19 . A method as recited in claim 1 , wherein the detecting of the anomalies and the detecting of the threat are performed by using Apache Storm or Apache Spark Streaming.

20 . A method as recited in claim 1 , wherein the detecting of the anomalies and the detecting of the threat are performed by using a data-parallel distributed processing engine.

21 . A method as recited in claim 1 , wherein the detecting of the anomalies and the detecting of the threat are performed by using Apache Spark.

22 . A method as recited in claim 1 , wherein the feature sets of event data are stored in a distributed file system, and wherein the detecting of the anomalies and the detecting of the threat are performed in a batch processing mode based on the stored feature sets.

23 . A method as recited in claim 1 , wherein the feature sets of event data are stored in a Hadoop Distributed File System (HDFS), and wherein the detecting of the anomalies and the detecting of the threat are performed in a batch processing mode based on the stored feature sets.

24 . A method as recited in claim 1 , wherein the detecting of the anomalies comprises performing at least one of: automated user behavior analysis or automated user-entity behavior analysis.

25 . A computer system comprising:

a processor;

a communication device, operatively coupled to the processor, through which to receive feature sets of event data originating from a plurality of data sources on a computer network;

an anomaly decision engine to detect network security anomalies based on the feature sets;

a threat decision engine to receive anomaly data indicative of the detected network security anomalies and to detect a network security threat based on the anomaly data; and

a user interface engine to generate a user interface and to cause the user interface to output threat data indicative of the detected network security threat.

26 . A computer system as recited in claim 25 , wherein:

the anomaly decision engine comprises a plurality of machine learning anomaly models, each including logic to detect a different type of anomaly; and

the threat decision engine comprises a plurality of machine learning threat models, each including logic to detect a different type of threat.

27 . A computer system as recited in claim 25 , wherein detection of the anomalies and detection of the threat are performed in real time as the event data are received from the plurality of data sources;

the computer system further comprising:

a batch anomaly detection engine to detect, in batch processing mode, additional network security anomalies based on stored feature sets of event data originating from the plurality of data sources on the computer network; and

a batch threat detection engine to detect, in batch processing mode, an additional network security threat based on the detected additional network security anomalies.

28 . A non-transitory machine-readable storage medium for use in a processing system, the non-transitory machine-readable storage medium storing instructions, an execution of which in the processing system causes the processing system to perform operations comprising:

inputting, to an anomaly decision engine, feature sets of event data originating from a plurality of data sources on a computer network;

detecting, by the anomaly decision engine, network security anomalies based on the feature sets;

outputting, to a threat decision engine, anomaly data indicative of the detected network security anomalies;

detecting, by the threat decision engine, a network security threat based on the anomaly data; and

outputting, via a user interface, threat data indicative of the detected network security threat.

29 . A non-transitory machine-readable storage medium as recited in claim 28 , wherein:

the anomaly decision engine comprises a plurality of machine learning anomaly models, each including logic to detect a different type of anomaly; and

the threat decision engine comprises a plurality of machine learning threat models, each including logic to detect a different type of threat.

30 . A non-transitory machine-readable storage medium as recited in claim 29 , the detecting of the anomalies and the detecting of the threat being performed in real time as the event data are received from the plurality of data sources;

the operations further comprising, in a batch processing mode:

detecting additional network security anomalies based on stored feature sets of event data originating from the plurality of data sources on the computer network; and

detecting an additional network security threat based on the detected additional network security anomalies.