Method and system for webpage classification and content delivery
In some aspects, the techniques described herein relate to a method including: receiving, at a processor, an uncrawled URL corresponding to a webpage; applying, by the processor, a webpage classification model to the uncrawled URL to determine a probability for a plurality of webpage classifications; assigning, by the processor, one or more labels to the uncrawled URL corresponding to one or more classifications of the plurality of webpage classifications that meet a threshold; and providing, by the processor, a content item to be displayed on the webpage based on the one or more labels.
1 . A method comprising:
receiving, by a device, information related to an uncrawled uniform resource locator (URL) for a webpage, the uncrawled URL being a URL for the webpage that has not been previously analyzed or indexed;
applying, by the device, a webpage classification model to the uncrawled URL information, and determining, based on the application of the webpage classification model, a probability for a plurality of webpage classifications, the webpage classification model being a student model trained via knowledge distillation using a set of URLs and associated soft labels output by a teacher model, the teacher model being trained on URLs and associated page content for a set of webpages;
analyzing, by the device, the plurality of webpage classifications, and identifying, based at least in part on the determined probability for each of the plurality of webpage classifications, a set of webpage classifications;
assigning, by the device, a set of labels to the uncrawled URL corresponding to the set of webpage classifications; and
providing, by the device, without crawling the webpage, a content item to be displayed on the webpage based on the assigned labels.
2 . The method of claim 1 , further comprising:
analyzing the set of webpage classifications; and
determining, based on the analysis of the set of webpage classifications, a set of categories, at least one of the set of webpage classifications corresponding to each category.
3 . The method of claim 2 , further comprising:
identifying, based on the analysis of the plurality of webpage classifications, a category from the set of categories; and
assigning a categorical label to the uncrawled URL.
4 . The method of claim 1 , further comprising:
receiving, over a network, the content item from a content creator; and
identifying, based on the assigned labels, the webpage, wherein the providing of the content item is based on the identification of the webpage via the assigned labels.
5 . The method of claim 4 , further comprising:
analyzing content of the content item, and determining the content to correspond to content of the webpage based on the assigned labels.
6 . The method of claim 1 , wherein the uncrawled URL information comprises data related to at least one of the URL and content of the webpage, the data for the URL corresponding to a domain and path of the URL.
7 . The method of claim 1 , wherein the identified set of webpage classifications each have a probability value at or above a threshold.
8 . The method of claim 1 , wherein the assignment of the set of labels occurs off-line.
9 . The method of claim 1 , wherein the webpage classification model is a type of model that corresponds to a type of the webpage.
10 . The method of claim 1 , wherein the set of labels correspond to layers of the webpage classification model.
11 . A non-transitory computer-readable storage medium tangibly storing thereon computer-executable instructions that when executed by a device, cause the device to perform a method comprising:
receiving, by the device, information related to an uncrawled uniform resource locator (URL) for a webpage, the uncrawled URL being a URL for the webpage that has not been previously analyzed or indexed;
applying, by the device, a webpage classification model to the uncrawled URL information, and determining, based on the application of the webpage classification model, a probability for a plurality of webpage classifications, the webpage classification model being a student model trained via knowledge distillation using a set of URLs and associated soft labels output by a teacher model, the teacher model being trained on URLs and associated page content for a set of webpages;
analyzing, by the device, the plurality of webpage classifications, and identifying, based at least in part on the determined probability for each of the plurality of webpage classifications, a set of webpage classifications;
assigning, by the device, a set of labels to the uncrawled URL corresponding to the set of webpage classifications; and
providing, by the device, without crawling the webpage, a content item to be displayed on the webpage based on the assigned labels.
12 . The non-transitory computer-readable storage medium of claim 11 , further comprising:
analyzing the set of webpage classifications; and
determining, based on the analysis of the set of webpage classifications, a set of categories, at least one of the set of webpage classifications corresponding to each category.
13 . The non-transitory computer-readable storage medium of claim 12 , further comprising:
identifying, based on the analysis of the plurality of webpage classifications, a category from the set of categories; and
assigning a categorical label to the uncrawled URL.
14 . The non-transitory computer-readable storage medium of claim 11 , further comprising:
receiving, over a network, the content item from a content creator;
analyzing content of the content item, and determining the content to correspond to content of the webpage based on the assigned labels; and
identifying, based on the assigned labels, the webpage, wherein the providing of the content item is based on the identification of the webpage via the assigned labels.
15 . The non-transitory computer-readable storage medium of claim 11 , wherein the uncrawled URL information comprises data related to at least one of the URL and content of the webpage, the data for the URL corresponding to a domain and path of the URL.
16 . A device comprising:
a processor configured to:
receive information related to an uncrawled uniform resource locator (URL) for a webpage, the uncrawled URL being a URL for the webpage that has not been previously analyzed or indexed;
apply a webpage classification model to the uncrawled URL information, and determine, based on the application of the webpage classification model, a probability for a plurality of webpage classifications, the webpage classification model being a student model trained via knowledge distillation using a set of URLs and associated soft labels output by a teacher model, the teacher model being trained on URLs and associated page content for a set of webpages;
analyze the plurality of webpage classifications, and identify, based at least in part on the determined probability for each of the plurality of webpage classifications, a set of webpage classifications;
assign a set of labels to the uncrawled URL corresponding to the set of webpage classifications; and
provide, without crawling the webpage, a content item to be displayed on the webpage based on the assigned labels.
17 . The device of claim 16 , wherein the processor is further configured to:
analyze the set of webpage classifications; and
determine, based on the analysis of the set of webpage classifications, a set of categories, at least one of the set of webpage classifications corresponding to each category.
18 . The device of claim 17 , wherein the processor is further configured to:
identify, based on the analysis of the plurality of webpage classifications, a category from the set of categories; and
assign a categorical label to the uncrawled URL.
19 . The device of claim 16 , wherein the processor is further configured to:
receive, over a network, the content item from a content creator;
analyze content of the content item, and determining the content to correspond to content of the webpage based on the assigned labels; and
identify, based on the assigned labels, the webpage, wherein the providing of the content item is based on the identification of the webpage via the assigned labels.
20 . The device of claim 16 , wherein the uncrawled URL information comprises data related to at least one of the URL and content of the webpage, the data for the URL corresponding to a domain and path of the URL.