SYSTEMS AND METHODS FOR CATEGORIZING DOMAINS USING ARTIFICIAL INTELLIGENCE
In an embodiment, a set of labeled training data that includes indicators of webpages is received. Each indicated webpage is labeled with one or more categories that were determined for the webpage by a human reviewer. Features, such as text and scripts, are extracted from each indicated webpage, and are used along with the labels to train a classifier to predict one or more categories for a webpage based on the features of the webpage. The trained classifier may be used to associate one or more categories with each domain of a plurality of domains given the categories predicted for some or all of the webpages associated with the domain. A list of domains and associated categories may be used for a variety of purposes including search engine optimization and content filtering.
1 . A method for training a classifier comprising:
receiving a training set of webpages by a computing device, wherein each webpage in the training set is labeled with a category of a first plurality of categories;
retrieving a second plurality of categories stored on the computing device by the computing device, wherein the second plurality of categories has fewer categories than the first plurality of categories;
for each webpage of the training set of webpages that is labeled with a category that is in the first plurality of categories but not in the second plurality of categories, replacing the category that the webpage is labeled with using a category from the second plurality of categories by the computing device;
for each webpage of the training set of webpages, extracting one or more features from the webpage by the computing device, wherein the one or more features comprise script features; and
for each webpage of the training set of webpages, training a neural network classifier using the one or more extracted features and the category that the webpage is labeled with by the computing device.
2 . The method of claim 1 , wherein each category of the first plurality of categories and the second plurality of categories relates to a topic or a subject, and the categories of the second plurality of categories are more general and/or generic than the categories of the first plurality of categories.
3 . The method of claim 1 , wherein the one or more features further comprise video features or image features.
4 . The method of claim 1 , further comprising:
for each domain of a plurality of domains:
retrieving a set of webpages from the domain by the computing device;
for each webpage of the set of webpages:
extracting one or more features from the webpage of the set of webpages by the computing device; and
associating a category of the second plurality of categories with the webpage using the neural network classifier and the one or more features extracted from the webpage by the computing device.
5 . The method of claim 4 , further comprising:
for each domain of the plurality of domains, associating a category of the second plurality of categories with the domain based on the category associated with each webpage of the set of webpages from the domain.
6 . The method of claim 5 , wherein associating a category of the second plurality of categories with the domain based on the category associated with each webpage of the set of webpages from the domain comprises:
determining each category associated with more than a threshold percentage of webpages of the set of webpages; and
associating the determined categories with the domain.
7 . The method of claim 6 , wherein each category is associated with a different threshold percentage and is identified by the neural network.
8 . A system for training a classifier comprising:
at least one processor; and
a computer-readable medium storing computer executable instructions stored therefore that when executed by the at least one processor cause the system to:
receive a training set of webpages, wherein each webpage in the training set is labeled with a category of a first plurality of categories;
retrieve a second plurality of categories stored on the system, wherein the second plurality of categories has fewer categories than the first plurality of categories;
for each webpage of the training set of webpages that is labeled with a category that is in the first plurality of categories but not in the second plurality of categories, replacing the category that the webpage is labeled with using a category from the second plurality of categories;
for each webpage of the training set of webpages, extract one or more features from the webpage, wherein the one or more features comprise script features; and
for each webpage of the training set of webpages, train a neural network classifier using the one or more extracted features and the category that the webpage is labeled with.
9 . The system of claim 8 , wherein each category of the first plurality of categories and the second plurality of categories relates to a topic or a subject, and the categories of the second plurality of categories are more general than the categories of the first plurality of categories.
10 . The system of claim 8 , wherein the one or more features further comprise video features or image features.
11 . The system of claim 8 , further comprising computer executable instructions stored therefore that when executed by the at least one processor cause the system to:
for each domain of a plurality of domains:
retrieve a set of webpages from the domain;
for each webpage of the set of webpages:
extract one or more features from the webpage of the set of webpages; and
associate a category of the second plurality of categories with the webpage using the neural network classifier and the one or more features extracted from the webpage.
12 . The system of claim 11 , further comprising computer executable instructions stored therefore that when executed by the at least one processor cause the system to:
for each domain of the plurality of domains, associate a category of the second plurality of categories with the domain based on the category associated with each webpage of the set of webpages from the domain.
13 . The system of claim 12 , wherein associating a category of the second plurality of categories with the domain based on the category associated with each webpage of the set of webpages from the domain comprises:
determining each category associated with more than a threshold percentage of webpages of the set of webpages; and
associating the determined categories with the domain.
14 . The system of claim 13 , wherein each category is associated with a different threshold percentage and is identified by the neural network.
15 . A non-transitory computer-readable medium storing computer executable instructions stored therefore that when executed by at least one processor a system to:
receive a training set of webpages, wherein each webpage in the training set is labeled with a category of a first plurality of categories;
retrieve a second plurality of categories stored on the system, wherein the second plurality of categories has fewer categories than the first plurality of categories;
for each webpage of the training set of webpages that is labeled with a category that is in the first plurality of categories but not in the second plurality of categories, replacing the category that the webpage is labeled with using a category from the second plurality of categories;
for each webpage of the training set of webpages, extract one or more features from the webpage, wherein the one or more features comprise script features; and
for each webpage of the training set of webpages, train a neural network classifier using the one or more extracted features and the category that the webpage is labeled with.
16 . The non-transitory computer-readable medium of claim 15 , wherein each category of the first plurality of categories and the second plurality of categories relates to a topic or a subject, and the categories of the second plurality of categories are more general and/or generic than the categories of the first plurality of categories.
17 . The non-transitory computer-readable medium of claim 15 , wherein the one or more features further comprise video features or image features.
18 . The non-transitory computer-readable medium of claim 15 , further comprising computer executable instructions stored therefore that when executed by the at least one processor cause the system to:
for each domain of a plurality of domains:
retrieve a set of webpages from the domain;
for each webpage of the set of webpages:
extract one or more features from the webpage of the set of webpages; and
associate a category of the second plurality of categories with the webpage using the neural network classifier and the one or more features extracted from the webpage.
19 . The non-transitory computer-readable medium of claim 18 , further comprising computer executable instructions stored therefore that when executed by the at least one processor cause the system to:
for each domain of the plurality of domains, associate a category of the second plurality of categories with the domain based on the category associated with each webpage of the set of webpages from the domain.
20 . The non-transitory computer-readable medium of claim 19 , wherein associating a category of the second plurality of categories with the domain based on the category associated with each webpage of the set of webpages from the domain comprises:
determining each category associated with more than a threshold percentage of webpages of the set of webpages; and
associating the determined categories with the domain.