Artificial intelligence for product data extraction
A computer system and method may be used to generate a product catalog from one or more websites. One or more product pages on the websites may be identified and parsed. Attribute information may be identified in each page. Moreover, one or more automated interactions may be performed to generate page variations and identify attribute values. The attribute information and attribute values may be stored as structured data in a database.
1. A computer-implemented method for extracting content from a web page comprising:
crawling a website to identify a plurality of product pages, the product pages comprising base product pages comprising data about a product;
applying an unsupervised content extraction model to the product pages to identify a first set of data patterns for extracting product attributes and to distinguish product attributes from non-product information;
filtering the first set of data patterns for extracting product attributes to create a refined second set of data patterns for extracting product attributes;
applying the second set of data patterns for extracting raw product attributes to the product pages;
identifying a plurality of interface elements, comprising menus and/or buttons, on the product pages;
applying an automated process to systematically activate the plurality of interface elements on the product pages, comprising menus and/or buttons, to generate respective product page variations;
generating differences between the product page variations, generated by systematically activating the plurality of interface elements on the product pages, comprising menus and/or buttons, and the base product pages and analyzing the generated differences to identify a plurality of raw product attribute values for each of the plurality of product attributes;
normalizing the plurality of product attributes and normalizing the plurality of raw product attribute values, identified by analyzing the generated differences between the product page variations and the base product page, to a master list of attributes by identifying corresponding standardized attribute values, to create consistency across multiple websites;
storing in a searchable database one or more product identifiers, product attributes, and product attribute values, identified by analyzing the generated differences between the product page variations and the base product page, and
enabling a user to search for and review product variations using the searchable database.
2. The computer-implemented method of claim 1 , further comprising:
crawling the web site to visit a plurality of linked web pages and extract a plurality of URLs from the linked web pages;
clustering the extracted URLs based on similarity to generate a plurality of URL clusters; generating a plurality of URL templates based on the URL clusters;
tagging a first set of URL templates as product page URLs;
prioritizing the product page URLs while crawling the website to identify the plurality of product pages.
3. The computer-implemented method of claim 1 , further comprising:
crawling the website to identify the plurality of product pages by using a reinforcement learning algorithm, where the reinforcement learning algorithm is rewarded for crawling a product page and penalized for visiting a non-product page.
4. The computer-implemented method of claim 1 , wherein the unsupervised content extraction model is a machine learning model.
5. The computer-implemented method of claim 4 , wherein the unsupervised content extraction model is trained to accept as input a plurality of HTML elements and select one of the HTML elements as a product attribute.
6. The computer-implemented method of claim 5 , further comprising:
extracting a plurality of HTML elements from a web page and extracting features from each of the HTML elements;
inputting the extracted features to the unsupervised content extraction model to select one of the plurality of HTML elements as a product attribute.
7. The computer-implemented method of claim 1 , wherein the first set of data patterns and the refined second set of data patterns for extracting product attributes comprise regular expressions.
8. The computer-implemented method of claim 1 , further comprising:
providing a master list of product attributes and attribute values and normalizing the product attributes and attributes values to the master list.
9. The computer-implemented method of claim 1 , wherein the searchable database comprises a graph, the graph comprising nodes representing products and edges representing common attributes between connected nodes.
10. The computer-implemented method of claim 1 , further comprising:
training an embedding generation model to generate tensors for each product in the searchable database, where the distance between tensors represents the similarity of the products in at least one dimension.
11. A non-transitory computer-readable medium comprising instructions that when executed by a processor, cause the computer device to perform operations comprising:
crawling a website to identify a plurality of product pages, the product pages comprising base product pages comprising data about a product;
applying an unsupervised content extraction model to the product pages to generate a first set of patterns for extracting product attributes;
filtering the first set of patterns for extracting product attributes to create a refined, second set of data patterns for extracting product attributes;
applying the second set of patterns for extracting product attributes to the product pages;
identifying a plurality of interface elements, comprising menus and/or buttons, on the product pages;
applying an automated process to systematically activate the plurality of interface elements on the product paces, comprising menus and/or buttons, to generate product page variations;
generating differences between the product page variations, generated by systematically activating the plurality of interface elements on the product pages, comprising menus and/or buttons, and the base product pages and analyzing the generated differences to identify a plurality of raw product attribute values for each of the plurality of product attributes;
normalizing the plurality of product attributes and normalizing the plurality of raw product attribute values, identified by analyzing the generated differences between the product page variations and the base product page, to a master set of attributes by identifying corresponding standardized attribute values;
storing in a searchable database one or more product identifiers, product attributes, and attribute values, identified by analyzing the generated differences between the product page variations and the base product page; and
enabling a user to search for and review product variations using the searchable database.
12. The non-transitory computer-readable medium of claim 11 , further comprising instructions for:
crawling the web site to visit a plurality of linked web pages and extract a plurality of URLs from the linked web pages;
clustering the extracted URLs based on similarity to generate a plurality of URL clusters;
generating a plurality of URL templates based on the URL clusters;
tagging a first set of URL templates as product page URLs;
prioritizing the product page URLs while crawling the website to identify the plurality of product pages.
13. The non-transitory computer-readable medium of claim 11 , further comprising instructions for:
crawling the website to identify the plurality of product pages by using a reinforcement learning algorithm, where the reinforcement learning algorithm is rewarded for crawling a product page and penalized for visiting a non-product page.
14. The non-transitory computer-readable medium of claim 11 , wherein the unsupervised content extraction model is a machine learning model.
15. The non-transitory computer-readable medium of claim 14 , wherein the unsupervised content extraction model is trained to accept as input a plurality of HTML elements and select one of the HTML elements as a product attribute.
16. The non-transitory computer-readable medium of claim 15 , further comprising instructions for:
extracting a plurality of HTML elements from a web page and extracting features from each of the HTML elements;
inputting the extracted features to the unsupervised content extraction model to select one of the plurality of HTML elements as a product attribute.
17. The non-transitory computer-readable medium of claim 11 , wherein the first set of patterns and second set of patterns for extracting product attributes comprise regular expressions.
18. The non-transitory computer-readable medium of claim 11 , further comprising instructions for:
providing a master list of product attributes and attribute values and normalizing the product attributes and attributes values to the master list.
19. The non-transitory computer-readable medium of claim 11 , wherein the searchable database comprises a graph, the graph comprising nodes representing products and edges representing common attributes between connected nodes.
20. The non-transitory computer-readable medium of claim 11 , further comprising instructions for:
training an embedding generation model to generate tensors for each product in the searchable database, where the distance between tensors represents the similarity of the products in at least one dimension.