Systems and method for categorizing domains using artificial intelligence
In an embodiment, a set of labeled training data that includes indicators of webpages is received. Each indicated webpage is labeled with one or more categories that were determined for the webpage by a human reviewer. Features, such as text and scripts, are extracted from each indicated webpage, and are used along with the labels to train a classifier to predict one or more categories for a webpage based on the features of the webpage. The trained classifier may be used to associate one or more categories with each domain of a plurality of domains given the categories predicted for some or all of the webpages associated with the domain. A list of domains and associated categories may be used for a variety of purposes including search engine optimization and content filtering.
1. A method for associating categories with domains comprising:
receiving a list of domains by a computing device, wherein each domain in the list of domains is associated with a set of webpages published at the domain;
receiving a plurality of categories from an entity by the computing device, wherein each category of the plurality of categories comprises a topic that the entity desires to restrict access to for one or more users;
receiving a classifier by the computing device;
for each domain of the list of domains:
retrieving the set of webpages published at the domain by the computing device;
for each webpage of the set of webpages:
extracting a plurality of features from the webpage of the set of webpages by the computing device, wherein the plurality of features comprise script features, and wherein the script features comprise portions of scripts found in the webpage of the set of webpages; and
associating one or more categories of the plurality of categories with the webpage using the classifier and the plurality of features extracted from the webpage by the computing device; and
for each category of the plurality of categories:
determining the percentage of webpages of the set of webpages published at the domain that were associated with the category by the classifier;
determining the threshold percentage set for the category, wherein each category was set with a different threshold percentage;
determining that the percentage of webpages is greater than the threshold percentage for the category; and
in response to determining that the percentage of webpages is greater than the threshold percentage for the category, associating the category with the domain.
2. The method of claim 1 , wherein the plurality of features further comprise text features, image features, and video features.
3. The method of claim 1 , further comprising:
receiving an indication of the creation of a new domain, wherein the new domain is not in the list of domains;
determining that a number of webpages published at the new domain exceeds a threshold number; and
in response to the determination, adding the new domain to the list of domains.
4. The method of claim 1 , further comprising:
receiving indications of a training set of webpages by the computing device, wherein each webpage in the training set is associated with one or more categories of the plurality of categories;
for each webpage of the training set of webpages, extracting one or more features from the webpage by the computing device; and
for each webpage of the training set of webpages, training the classifier using the one or more extracted features and the one or more categories associated with the webpage by the computing device.
5. The method of claim 1 , further comprising using the list of domains and associated one or more categories to control user access to the set of webpages associated with each domain of the list of domains.
6. A system for associating categories with domains comprising:
at least one processor; and
a computer-readable medium storing computer executable instructions stored therefore that when executed by the at least one processor cause the system to:
receive a list of domains, wherein each domain in the list of domains is associated with a set of webpages published at the domain;
receive a plurality of categories, wherein each category of the plurality of categories comprises a topic that an entity desires to restrict access to for one or more users;
receive a classifier;
for each domain of the list of domains:
retrieve set of webpages published at the domain;
for each webpage of the set of webpages:
extract a plurality of features from the webpage of the set of webpages, wherein the plurality of features comprise script features, and wherein the script features comprise portions of scripts found in the webpage of the set of webpages; and
associate one or more categories of the plurality of categories with the webpage using the classifier and the plurality of features extracted from the webpage; and
for each category of the plurality of categories:
determine the percentage of webpages of the set of webpages published at the domain that were associated with the category by the classifier;
determine the threshold percentage set for the category, wherein each category was set with a different threshold percentage;
determine that the percentage of webpages is greater than the threshold percentage for the category; and
in response to determining that the percentage of webpages is greater than the threshold percentage for the category, associate the category with the domain.
7. The system of claim 6 , wherein the plurality of features further comprise text features, image features, and video features.
8. The system of claim 6 , wherein the classifier is a neural network.
9. The system of claim 6 , further comprising computer executable instructions stored therefore that when executed by the at least one processor cause the system to:
receive indications of a training set of webpages, wherein each webpage in the training set is associated with one or more categories of the plurality of categories;
for each webpage of the training set of webpages, extract one or more features from the webpage; and
for each webpage of the training set of webpages, train the classifier using the one or more extracted features and the one or more categories associated with the webpage.
10. The system of claim 6 , further comprising computer executable instructions stored therefore that when executed by the at least one processor cause the system to use the list of domains and associated one or more categories to control user access to the set of webpages associated with each domain of the list of domains.
11. A non-transitory computer-readable medium storing computer executable instructions stored therefore that when executed by at least one processor cause the at least one processor to:
receive a list of domains, wherein each domain in the list of domains is associated with a set of webpages published at the domain;
receive a plurality of categories, wherein each category of the plurality of categories comprises a topic that an entity desires to restrict access to for one or more users;
receive a classifier;
for each domain of the list of domains:
retrieve the set of webpages published at the domain;
for each webpage of the set of webpages:
extract a plurality of features from the webpage of the set of webpages, wherein the plurality of features comprise script features, and wherein the script features comprise portions of scripts found in the webpage of the set of webpages; and
associate one or more categories of the plurality of categories with the webpage using the classifier and the plurality of features extracted from the webpage; and
for each category of the plurality of categories:
determine the percentage of webpages of the set of webpages published at the domain that were associated with the category by the classifier;
determine the threshold percentage set for the category, wherein each category was set with a different threshold percentage;
determine that the percentage of webpages is greater than the threshold percentage for the category; and
in response to determining that the percentage of webpages is greater than the threshold percentage for the category, associate the category with the domain.
12. The computer-readable medium of claim 11 , wherein the features comprise text features, image features, and script features.
13. The computer-readable medium of claim 11 , wherein the classifier is a neural network.
14. The computer-readable medium of claim 11 , further comprising computer executable instructions stored therefore that when executed by the at least one processor cause the system to:
receive indications of a training set of webpages, wherein each webpage in the training set is associated with one or more categories of the plurality of categories;
for each webpage of the training set of webpages, extract one or more features from the webpage; and
for each webpage of the training set of webpages, train the classifier using the one or more extracted features and the one or more categories associated with the webpage.