System and method for the automated detection and prediction of online threats
A method for monitoring online security threats comprising of a machine-learning service that receives data related to a plurality of features related to internet traffic metrics, the service then processes said data by performing operations selected from among: an operation of ranking at least one feature, an operation of classifying at least one feature, an operation of predicting at least one feature, and an operation of clustering at least one feature, and as a result the machine learning service outputs metrics that aid in the detection, identification, and prediction of an attack.
1. A method for alerting a client system of a potential malware threat, comprising:
receiving a first set of network traffic data of a source system;
extracting from the first set of network traffic data: a first Internet Protocol (IP) address, timestamps of identified attacks, a Universal Record Locator (URL), and a domain name;
using the first IP address to determine a network queue;
using the timestamps of the identified attacks on the network queue to determine queuing based features, wherein the queuing based features include an infection arrival rate and an infection service rate;
using the URL to determine URL based features, wherein the URL based features include at least one of an entropy of the URL and a length of the URL;
using the domain name to determine domain based features, wherein the domain based features include historical replication of Domain Name System (DNS) data for the first IP address or the domain name;
using the first IP address to determine a network prefix Classless Inter-Domain Routing (CIDR) and an Autonomous System Number (ASN);
using the network prefix (CIDR) and the Autonomous System Number (ASN) to determine Boarder Gateway Protocol (BGP) based features;
wherein the BGP based features includes CIDR size, ASN size, BGP route length, and a flag indicating whether the first IP address belongs to a hijacked route or prefix;
aggregating the queuing based features, the URL based features, the BGP based features, and the domain based features to generate a trained model;
receiving a second set of network traffic data of a target system;
wherein the second set of network traffic data comprises a second IP address;
applying the trained model to the second set of traffic data to score a threat level of the second IP address;
alerting the client system of the threat level;
wherein the first set of network traffic data is agnostic with respect to the second set of network traffic data;
wherein at least one of the first set of network traffic data and the second set of network traffic data includes at least one of the following: Threat feeds, DNS Logs, Hypertext Transfer Protocol (HTTP) Logs, Hypertext Markup Language (HTML) Webpages, Malware Payload, Malware Sandbox, System Logging Protocols (Syslogs), Packet Capture Application Programming Interfaces (PCAPs), and Regional Internet Registry (RIR) information; and
wherein the source system is different from the client system.
2. The method of claim 1 , further comprising aggregating and processing multiple threat feeds from additional clients, and clustering meta information from the multiple threat feeds according to network size.
3. The method of claim 2 wherein the meta information comprises at least one of the following: a time series, URLs, Domains, attack category, Threat Feed, Payload Analysis, BGP Extraction.
4. The method of claim 2 wherein the meta information comprises at least one of the following: new IP assignments, new IP allocations, new IP re-assignments, and new IP re-allocations.
5. The method of claim 2 wherein the clustering of meta information is based on at least one of contact entropy based features and Passive DNS (pDNS) based features.