Predictive modeling of domain names using web-linking characteristics
A method evaluates whether a web domain is malicious. The method forms a feature vector, including data from web crawling. The features may include: whether the domain is cached from web crawling; the number of unique publicly accessible URIs hosted on the domain; the number of backlinks referencing the domain; the number of unique domain names in referring backlinks; the number of unique IP addresses in the referring backlinks; the number of unique IP address groups in the referring backlinks; and the proportion of hyperlinks to the domain from popular websites. For multiple classifiers, the method computes a probability that the domain is malicious. Each classifier is a decision tree constructed according to a subset of features and a subset of sample feature vectors. The method combines the individual probabilities to form an overall probability and returns the computed overall probability to the client.
1. A method for classifying domains, performed at a computer system having a plurality of classifiers, one or more processors and memory storing one or more programs configured for execution by the one or more processors, the method comprising:
receiving, from a client, a request to evaluate whether a web domain is malicious, the request including an identifier of the web domain;
forming a feature vector v=[v 1 , v 2 , . . . , v n ] whose components include:
a number of unique publicly accessible URIs that were found to be hosted on the domain in a previous web crawling;
a count of non-unique hyperlinks from a homepage of the domain;
a count of non-unique image-based hyperlinks from the homepage of the domain;
a count of non-unique text-based hyperlinks from the home page of the domain;
a count of non-unique hyperlinks, from a home page of the domain, that include the string “https://”;
a count of non-unique nofollow hyperlinks from the homepage of the domain;
a count of non-unique nofollow backlinks referencing the domain; and
a relative proportion of hyperlinks to the domain from websites that are designated as popular based on user access;
for each classifier of the plurality of classifiers, computing a respective probability that the web domain is malicious, wherein each classifier is a respective decision tree constructed according to a respective distinct subset of the plurality of features and according to a respective distinct subset of predetermined sample feature vectors corresponding to web domains known to be benign or known to be malicious;
computing an overall probability that the web domain is malicious using a weighted sum of the plurality of computed probabilities; and
returning the overall probability to the client.
2. The method of claim 1 , wherein the identifier of the web domain is selected from the group consisting of a URL, a domain name, and an IP address.
3. The method of claim 1 , wherein the feature vector further includes one or more features selected from the group consisting of:
a status of whether the domain is present in a data cache from previous web crawling;
a count of backlinks referencing the domain;
a count of unique domain names in referring backlinks;
a count of unique IP addresses in the referring backlinks; and
a count of unique IP address groups in the referring backlinks.
4. The method of claim 3 , wherein each of the unique IP address groups consists of all IP addresses sharing a respective initial address portion.
5. The method of claim 4 , wherein each initial address portion of a unique IP address group consists of the first 24-bit segment of each IP address in the respective unique IP address group.
6. The method of claim 1 , wherein the previous web crawling populates the data cache recursively by querying hyperlinks associated with domains previously identified by the web crawling.
7. The method of claim 1 , wherein the feature vector includes one or more features extracted from domain registration data.
8. An electronic device, comprising:
one or more processors; and
memory storing one or more programs for execution by the one or more processors, the one or more programs including instructions for:
receiving, from a client, a request to evaluate whether a web domain is malicious, the request including an identifier of the web domain;
forming a feature vector v=[v 1 , v 2 , . . . , v n ] whose components include:
a number of unique publicly accessible URIs that were found to be hosted on the domain in the previous web crawling;
a count of non-unique hyperlinks from a homepage of the domain;
a count of non-unique image-based hyperlinks from the homepage of the domain;
a count of non-unique text-based hyperlinks from the home page of the domain;
a count of non-unique hyperlinks, from a home page of the domain, that include the string “https://”;
a count of non-unique nofollow hyperlinks from the homepage of the domain;
a count of non-unique nofollow backlinks referencing the domain; and
a relative proportion of hyperlinks to the domain from websites that are designated as popular based on user access;
for each classifier of a plurality of classifiers, computing a respective probability that the web domain is malicious, wherein each classifier is a respective decision tree constructed according to a respective distinct subset of the plurality of features and according to a respective distinct subset of predetermined sample feature vectors corresponding to web domains known to be benign or known to be malicious;
computing an overall probability that the web domain is malicious using a weighted sum of the plurality of computed probabilities; and
returning the overall probability to the client.
9. The electronic device of claim 8 , wherein the identifier of the web domain is selected from the group consisting of a URL, a domain name, and an IP address.
10. The electronic device of claim 8 , wherein the feature vector further includes one or more features selected from the group consisting of:
a status of whether the domain is present in a data cache from previous web crawling;
a count of backlinks referencing the domain;
a count of unique domain names in referring backlinks;
a count of unique IP addresses in the referring backlinks; and
a count of unique IP address groups in the referring backlinks.
11. The electronic device of claim 10 , wherein each of the unique IP address groups consists of all IP addresses sharing a respective initial address portion.
12. The electronic device of claim 11 , wherein each initial address portion of a unique IP address group consists of the first 24-bit segment of each IP address in the respective unique IP address group.
13. The electronic device of claim 8 , wherein the previous web crawling populates the data cache recursively by querying hyperlinks associated with domains previously identified by the web crawling.
14. The electronic device of claim 8 , wherein the feature vector includes one or more features extracted from domain registration data.
15. A non-transitory computer-readable storage medium storing one or more programs for execution by one or more processors of an electronic device, the one or more programs including instructions for:
receiving, from a client, a request to evaluate whether a web domain is malicious, the request including an identifier of the web domain;
forming a feature vector v=[v 1 , v 2 , . . . , v n ] whose components include:
a number of unique publicly accessible URIs that were found to be hosted on the domain in the previous web crawling;
a relative proportion of hyperlinks to the domain from websites that are designated as popular based on user access;
a count of non-unique image-based hyperlinks from the homepage of the domain;
a count of non-unique text-based hyperlinks from the home page of the domain;
a count of non-unique hyperlinks, from a home page of the domain, that include the string “https://”;
a count of non-unique nofollow hyperlinks from the homepage of the domain; and
a count of non-unique nofollow backlinks referencing the domain;
for each classifier of a plurality of classifiers, computing a respective probability that the web domain is malicious, wherein each classifier is a respective decision tree constructed according to a respective distinct subset of the plurality of features and according to a respective distinct subset of predetermined sample feature vectors corresponding to web domains known to be benign or known to be malicious;
computing an overall probability that the web domain is malicious using a weighted sum of the plurality of computed probabilities; and
returning the overall probability to the client.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the identifier of the web domain is selected from the group consisting of a URL, a domain name, and an IP address.
17. The non-transitory computer-readable storage medium of claim 16 , wherein each of the unique IP address group consists of all IP addresses sharing a respective initial address portion.
18. The non-transitory computer-readable storage medium of claim 17 , wherein each initial address portion of a unique IP address group consists of the first 24-bit segment of each IP address in the respective unique IP address group.
19. The non-transitory computer-readable storage medium of claim 15 , wherein the previous web crawling populates the data cache recursively by querying hyperlinks associated with domains previously identified by the web crawling.
20. The non-transitory computer-readable storage medium of claim 15 , wherein the feature vector includes one or more features extracted from domain registration data.