Scalable incident-response and forensics toolkit
Techniques for performing data analytics using anomaly detection systems and methods are disclosed. The anomaly detection system provides an incident response and monitoring solution, built for distributed processing, that streamlines cyber defense by unifying datasets, via a data translator, from sensors and tools into a uniform schema to provide real-time anomaly detection, via an anomaly detection system that may prevent malware from establishing a foothold on the network. The anomaly detection system may allow for the scalability to provide large-scale data aggregation and anomaly detection without compromising performance. The anomaly detection system may use a distributed architecture to support advanced cyber threat detection across large datasets in real-time for monitoring and rapid incident response. The anomaly detection system may leverage open protocols and interfaces to promote third-party support for development and interoperability.
1. A method comprising:
receiving, by processing circuitry, a first data set in a first format, the first data set comprises at least one of network traffic, device logs or sensor information;
formulating, by the processing circuitry, one or more queries to a data application requesting a second data set in a second format different than the first format, the one or more queries in the second format;
sending, by the processing circuitry, the one or more queries to the data application;
receiving, by the processing circuitry, the second data set in the second format in response to the one or more queries;
converting, by the processing circuitry, the first data set and the second data set into a shared format; and
analyzing, by the processing circuitry, the first data set and the second data set in the shared format to detect anomalies.
2. The method of claim 1 , further comprising storing the first data set and the second data set in a shared open format.
3. The method of claim 1 , wherein analyzing the first data set and the second data set in the shared format is based on a random forest technique.
4. The method of claim 1 , wherein analyzing the first data set and the second data set in the shared format is based on a support vector machine technique.
5. The method of claim 1 , further comprising:
splitting the first data set and the second data set into a plurality of partitions;
sending each of the plurality of partitions to a different one of a plurality of distributed computing nodes,
wherein analyzing the first data set and the second data set in the shared format is performed in parallel by the plurality of distributed computing nodes.
6. The method of claim 1 further comprising providing, by the processing circuitry, a report based on the analysis of the first data set and the second data set in the shared format.
7. The method of claim 6 , wherein providing the report is via a secure web interface.
8. A computing device, comprising:
one or more processors; and
a computer-readable storage device communicatively coupled to the one or more processors, wherein the computer-readable storage device stores instructions that, when executed by the one or more processors, cause the one or more processors to:
receive a first data set in a first format, the first data set comprises at least one of network traffic, device logs or sensor information;
formulate one or more queries to a data application requesting a second data set in a second format different than the first format, the one or more queries in the second format;
send the one or more queries to the data application;
receive the second data set in the second format in response to the one or more queries;
convert the first data set and the second data set into a shared format; and
analyze the first data set and the second data set in the shared format to detect anomalies.
9. The computing device of claim 8 , wherein analyzing the first data set and the second data set in the shared format is based on a random forest technique or a support vector machine technique.
10. The computing device of claim 8 , wherein the computer-readable storage device stores further instructions that, when executed by the one or more processors, cause the one or more processors to:
split the first data set and the second data set into a plurality of partitions;
send each of the plurality of partitions to a different one of a plurality of distributed computing nodes,
wherein analyzing the first data set and the second data set in the shared format is performed in parallel by the plurality of distributed computing nodes.
11. The computing device of claim 8 , wherein the computer-readable storage device stores further instructions that, when executed by the one or more processors, cause the one or more processors to convert the one or more queries from the first format to the second format.
12. A non-transitory computer-readable storage medium stores instructions that,
when executed, cause one or more processors to:
receive a first data set in a first format, the first data set comprises at least one of network traffic, device logs or sensor information;
formulate one or more queries to a data application requesting a second data set in a second format different than the first format, the one or more queries in the second format;
send the one or more queries to the data application;
receive the second data set in the second format in response to the one or more queries;
convert the first data set and the second data set into a shared format; and
analyze the first data set and the second data set in the shared format to detect anomalies.
13. The non-transitory computer-readable storage medium of claim 12 , wherein analyzing the first data set and the second data set in the shared format is based on a random forest technique or a support vector machine technique.
14. The non-transitory computer-readable storage medium of claim 12 , wherein the instructions, when executed, further cause the one or more processors to:
split the first data set and the second data set into a plurality of partitions;
send each of the plurality of partitions to a different one of a plurality of distributed computing nodes,
wherein analyzing the first data set and the second data set in the shared format is performed in parallel by the plurality of distributed computing nodes.
15. The non-transitory computer-readable storage medium of claim 12 , wherein the instructions, when executed, further cause the one or more processors to convert the one or more queries from the first format to the second format.