Product page classification
In one embodiment, a method includes extracting a document object model (DOM) for a content page. The DOM comprises a hierarchical tree-based data structure. The method also includes traversing the DOM to identify, in the content page, candidate features of a product page. The method further includes for each of the candidate features, determine candidate feature attributes based on a respective context of the candidate feature in the DOM. The method additionally includes determining whether the content page qualifies as a product page based on whether the candidate features fulfill a required set of characteristics for a product page.
1. A method comprising:
by one or more processors, extracting a document object model (DOM) for a content page, wherein the DOM comprises a hierarchical tree-based data structure;
by the one or more processors, traversing the DOM to identify, in the content page, candidate features of a product page;
by the one or more processors, for each of the candidate features, determining candidate feature attributes based on a respective context of the candidate feature in the DOM, wherein the candidate features comprise candidate navigation elements that have a predetermined relationship with candidate product elements in the DOM;
by the one or more processors, interacting with at least one of the candidate navigation elements to determine whether it controls display of the candidate product elements in a manner that satisfies a predetermined display rule, wherein the predetermined display rule requires interactive functionality that allows selection of a desired plurality of the candidate product elements displayed in the content page;
by the one or more processors, determining whether the content page qualifies as either (1) a product page or (2) a featured products page based on whether the candidate features fulfill a required set of characteristics for a product page or a featured products page and whether the candidate navigation elements satisfy the required interactive functionality, wherein the required set of characteristics for the product page include that it displays products in a same category for comparison and the required set of characteristics for the featured products page include that it displays a sampling of representative products that do not share a same category; and
by the one or more processors, determining whether to crawl the content page based the determination as to whether it qualifies as a product page or a featured products page.
2. The method of claim 1 , wherein determining whether the content page qualifies as either a product page or a featured products page comprises determining whether the content page hosts one or more product listings.
3. The method of claim 1 , wherein the required set of characteristics for the product page comprises identification of at least one button associated with purchasing a product.
4. The method of claim 1 , wherein the required set of characteristics for the product page comprises identification of at least one zoom button associated with magnifying an image of a product.
5. The method of claim 1 , wherein the required set of characteristics for the product page comprises identification of more than a threshold number of available options associated with a product.
6. The method of claim 1 , wherein the required set of characteristics for the product page comprises determining whether a title associated with a product and a link associated with the product match a predetermined size ratio.
7. The method of claim 1 , further comprising, in response to determining that the content page qualifies as a product page, determining whether the product page qualifies as a product listing page based on whether the product page fulfills a predetermined set of characteristics for a product listing page.
8. The method of claim 1 , further comprising, in response to determining that the content page qualifies as a product page, determining whether the product page qualifies as a product details page based on whether the product page fulfills a predetermined set of characteristics for a product details page.
9. The method of claim 1 , wherein determining whether the candidate features fulfill the required set of characteristics for the product page comprises determining whether any of the candidate feature attributes are associated with displaying one or more products.
10. The method of claim 7 , wherein the required set of characteristics for the product page comprises identification of one or more features associated with iterating over a set of products.
11. The method of claim 7 , wherein the required set of characteristics for the product page comprises identification of more than a threshold number of features associated with a price.
12. The method of claim 1 , further comprising:
identifying a candidate feature corresponding to a title of the content page;
determining that the title of the content page is associated with a predetermined set of non-product keywords; and
determining that the content page is neither a product page or a featured products page.
13. The method of claim 1 , further comprising:
in response to determining that the content page does not qualify as either a product page or a featured products page, determining whether the content page qualifies as another page type based on whether the candidate features fulfill a required set of characteristics for the another page type.
14. The method of claim 1 , wherein determining whether the content page qualifies as either a product page or a featured products page is further based on an inbound link analysis of other content pages associated with a domain, wherein the content page is also associated with the domain.
15. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
extract a document object model (DOM) for a content page, wherein the DOM comprises a hierarchical tree-based data structure;
traverse the DOM to identify, in the content page, candidate features of a product page;
for each of the candidate features, determine candidate feature attributes based on a respective context of the candidate feature in the DOM, wherein the candidate features comprise candidate navigation elements that have a predetermined relationship with candidate product elements in the DOM;
interact with at least one of the candidate navigation elements to determine whether it controls display of the candidate product elements in a manner that satisfies a predetermined display rule, wherein the predetermined display rule requires interactive functionality that allows selection of a desired plurality of the candidate product elements displayed in the content page;
determine whether the content page qualifies as either (1) a product page or (2) a featured products page based on whether the candidate features fulfill a required set of characteristics for a product page or a featured products page and whether the candidate navigation elements satisfy the required interactive functionality, wherein the required set of characteristics for the product page include that it displays products in a same category for comparison and the required set of characteristics for the featured products page include that it displays a sampling of representative products that do not share a same category; and
determine whether to crawl the content page based the determination as to whether it qualifies as a product page or a featured products page.
16. The computer-readable non-transitory storage media of claim 15 , wherein determining whether the content page qualifies as either a product page or a featured products page comprises determining whether the content page hosts one or more product listings.
17. The computer-readable non-transitory storage media of claim 15 , wherein the required set of characteristics for the product page comprises identification of at least one button associated with purchasing a product.
18. The computer-readable non-transitory storage media of claim 15 , wherein the required set of characteristics for the product page comprises identification of at least one zoom button associated with magnifying an image of a product.
19. The computer-readable non-transitory storage media of claim 15 , wherein the required set of characteristics for the product page comprises identification of more than a threshold number of available options associated with a product.
20. A system comprising: one or more processors; and a non-transitory memory coupled to the one or more processors comprising instructions executable by the one or more processors, the one or more processors operable when executing the instructions to:
extract a document object model (DOM) for a content page, wherein the DOM comprises a hierarchical tree-based data structure;
traverse the DOM to identify, in the content page, candidate features of a product page;
for each of the candidate features, determine candidate feature attributes based on a respective context of the candidate feature in the DOM, wherein the candidate features comprise candidate navigation elements that have a predetermined relationship with candidate product elements in the DOM;
interact with at least one of the candidate navigation elements to determine whether it controls display of the candidate product elements in a manner that satisfies a predetermined display rule, wherein the predetermined display rule requires interactive functionality that allows selection of a desired plurality of the candidate product elements; and
determine whether the content page qualifies as either (1) a product page or (2) a featured products page based on whether the candidate features fulfill a required set of characteristics for a product page or a featured products page and whether the candidate navigation elements satisfy the required interactive functionality, wherein the required set of characteristics for the product page include that it displays products in a same category for comparison and the required set of characteristics for the featured products page include that it displays a sampling of representative products that do not share a same category; and
determine whether to crawl the content page based the determination as to whether it qualifies as a product page or a featured products page.