Automated extraction of data from web pages
Various embodiments provide techniques for automatically extracting data from web pages. Such extraction can take place without the use of a browser, and without necessarily rendering the entire web page. Thus, data extraction can be performed more efficiently and more quickly, while reducing the computing resources needed to perform such operations. In at least one embodiment, data extraction and translation are performed by automatically parsing structured data from visible and hidden elements of a web page.
1. A system, comprising:
at least one memory storing instructions; and
at least one processor, operatively connected to the at least one memory, and configured to perform a process including:
generating an API that is generalized across a plurality of websites by, for each website:
obtaining at least one reply transmitted by a web server associated with the website in response to a request transmitted to the web server;
based on the at least one reply, extracting data from the website;
applying an abstract syntax parser to the extracted data to generate an abstract syntax tree;
based on the abstract syntax tree, generating a translation between a format of the data from the website and a standardized API format; and
based on the translation, generating at least one API request that is specific to the website and that is executable to retrieve requested data from the website in the standardized API format; and
utilizing the generated API to obtain respective information from each of the plurality of websites by, in each case, transmitting the at least one generated API request specific to the website to the web server associated with the website, the information, in each case, pertaining to at least one aspect of a same product or service available on the plurality of websites.
2. The system of claim 1 , wherein extracting the data from the website includes:
obtaining a domain object model for the website;
applying a model trained by machine learning to identify targeted information from the domain object model; and
extracting the identified targeted information from the website.
3. The system of claim 2 , wherein the targeted information includes information having a predetermined structure.
4. The system of claim 1 , wherein extracting the data from the website includes, in each case:
analyzing script tag data within script tags included in the at least one reply, wherein the abstract syntax tree is generated based on the script tag data; and
traversing and interpreting the abstract syntax tree.
5. The system of claim 1 , wherein extracting the data from the website includes, in each case:
obtaining a domain object model for the website;
analyzing the domain object model to identify microdata of the website; and
extracting the microdata from the website.
6. The system of claim 1 , wherein the data from the website, in each case, is structured data.
7. The system of claim 1 , wherein:
the product or service is displayed on a website visited by a user device; and
the utilizing of the API is in response to the display on the website.
8. A computer-implemented method, comprising:
generating an API that is generalized across a plurality of websites by, for each website:
obtaining at least one reply transmitted by a web server associated with the website in response to a request transmitted to the web server;
based on the at least one reply, extracting data from the website; and
based on the extracted data, generating at least one API request that is specific to the website and that is executable to retrieve requested data from the website in a standardized API format; and
utilizing the generated API to obtain respective information from each of the plurality of websites by, in each case, transmitting the at least one generated API request specific to the website to the web server associated with the website, the information, in each case, pertaining to at least one aspect of a same product or service available on the plurality of websites.
9. The computer-implemented method of claim 8 , wherein extracting the data from the website includes:
analyzing script tag data within script tags included in the at least one reply.
10. The computer-implemented method of claim 8 , wherein extracting the data from the website includes:
obtaining a domain object model for the website;
analyzing the domain object model to identify microdata of the website; and
extracting the microdata from the website.
11. The computer-implemented method of claim 8 , wherein extracting the data from the website includes:
applying a model trained by machine learning to identify data from the website having a predetermined structure; and
extracting information having the predetermined structure from the website.
12. The computer-implemented method of claim 8 , wherein the data from the website, in each case, is structured data.
13. The computer-implemented method of claim 8 , wherein:
the product or service is displayed on a website visited by a user device; and
the utilizing of the API is in response to the display on the website.
14. A computer-implemented method, comprising:
generating an API that is generalized across a plurality of websites by, for each website:
obtaining at least one reply transmitted by a web server associated with the website in response to a request transmitted to the web server;
based on the at least one reply, extracting data from the website;
applying an abstract syntax parser to the extracted data to generate an abstract syntax tree;
traversing and interpreting the generated abstract syntax tree to determine a translation between a format of the data from the website and a standardized API format; and
based on the translation, generating at least one API request that is specific to the website and that is executable to retrieve requested data from the website in the standardized API format.
15. The computer-implemented method of claim 14 , wherein extracting the data from the website includes:
analyzing script tag data within script tags included in the at least one reply, wherein the abstract syntax tree is generated based on the script tag data; and
traversing and interpreting the abstract syntax tree.
16. The computer-implemented method of claim 14 , wherein extracting the data from the website includes:
obtaining a domain object model for the website;
analyzing the domain object model to identify microdata of the website; and
extracting the microdata from the website.
17. The computer-implemented method of claim 14 , further comprising:
utilizing the generated API to obtain respective information from each of the plurality of websites by, in each case, transmitting the at least one generated API request specific to the website to the web server associated with the website, the information, in each case, pertaining to at least one aspect of a same product or service available on the plurality of websites.
18. The computer-implemented method of claim 17 , wherein:
the product or service is displayed on a website visited by a user device; and
the utilizing of the API is in response to the display on the website.
19. The computer-implemented method of claim 14 , wherein the data from the website, in each case, is structured data.
20. The computer-implemented method of claim 14 , wherein extracting the data from the website includes:
applying a model trained by machine learning to identify data from the website having a predetermined structure; and
extracting information having the predetermined structure from the website.