Identifying search friendly web pages
A system for evaluating web pages for searchable content can be utilized to make an e-commerce search engine more effective by identifying pages with searchable value. In embodiments, a web page exhibiting “searchable value” is a page that provides useful information responsive to a user's query on an e-commerce search engine. One embodiment of a page evaluation system includes a web crawler module, a page analysis module, and a page index module. Web pages may be fetched and evaluated for various signals that indicate whether the web page exhibits searchable value. Examples of signals that may indicate that a web page lacks searchable value include a product page that does not have a product name nor a product image and a category page that does not have products listed in the page content. Web pages that lack searchable value may be excluded from subsequent e-commerce search results.
1. A computer-implemented method executed by a processor for identifying searchable content, the method comprising:
requesting a selected URL from a list of URLs to request, the selected URL belonging to a website;
receiving a set of site-specific rules for the website and following the site-specific rules to extract web page content from a web page of the website corresponding to the selected URL;
determining whether the web page content comprises at least one of a product detail content, a product listing content, a category page, and metadata;
determining whether the web page comprises a searchable value; and
if the web page does not comprise a searchable value, determining if the set of site-specific rules for the website are misconfigured and if so, updating the site-specific rules for the website and returning to the extracting of the web page content from the web page using the updated site-specific rules.
2. The method of claim 1 , wherein selecting to update the site-specific rules further comprises identifying that the site-specific rules are obsolete.
3. The method of claim 1 , wherein determining whether the web page comprises searchable value further comprises identifying a signal indicating whether the web page lacks searchable value.
4. The method of claim 3 , wherein the signal comprises identification of no product name and no product image in the product detail content.
5. The method of claim 3 , wherein the signal comprises identification of no product listing in the product listing content.
6. The method of claim 3 , wherein the signal comprises identification of no products listed in the category page.
7. The method of claim 3 , wherein the signal comprises identification of no product description and no product image in product detail content.