Automated extraction of data from web pages
Various embodiments provide techniques for automatically extracting data from web pages. Such extraction can take place without the use of a browser, and without necessarily rendering the entire web page. Thus, data extraction can be performed more efficiently and more quickly, while reducing the computing resources needed to perform such operations. In at least one embodiment, data extraction and translation are performed by automatically parsing structured data from visible and hidden elements of a web page.
1. A computer-implemented method for extracting data from a web page, comprising:
requesting and retrieving, by transmitting requests and receiving replies, respectively, at least one web page from a web server;
extracting structured data from the at least one web page;
based on the structured data, determining one or more requests, of the requests, contain parameters for displaying information;
based on a determination that the one or more requests contain the parameters for displaying the information, performing at least one of an extraction operation or a transform operation to obtain the information; and
generating and transmitting at least one request based on the one or more requests to obtain other information.
2. The computer-implemented method of claim 1 , further comprising:
mapping the other information into a consistent format.
3. The computer-implemented method of claim 1 , wherein performing at least one of the extraction operation or the transform operation to obtain the information comprises performing at least one task selected from a group consisting of:
parsing structured information in script tags;
extracting content from received HTML through DOM parsing; or
requesting the content returned in a readable structure.
4. The computer-implemented method of claim 1 , wherein extracting the structured data from the at least one web page comprises:
analyzing script tag data within script tags;
generating an abstract syntax tree based on the script tag data; and
traversing and interpreting the abstract syntax tree to extract the structured data.
5. The computer-implemented method of claim 1 , wherein extracting the structured data from the at least one web page comprises:
receiving a domain object model for the at least one web page;
analyzing the domain object model; and
based on an analysis result for the domain object model, locating microdata within the domain object model.
6. The computer-implemented method of claim 5 , wherein analyzing the domain object model comprises:
applying a model trained by machine learning, to analyze the domain object model.
7. The computer-implemented method of claim 1 , wherein extracting the structured data from the at least one web page comprises:
applying a model trained by machine learning, to identify the structured data to be extracted.
8. The computer-implemented method of claim 1 , wherein extracting the structured data from the at least one web page comprises:
analyzing a structured format by which the structured data has been organized;
based on an analysis result for the structured format, generating a format for future requests; and
requesting the other information from the web server using the format.
9. A non-transitory computer-readable medium for extracting data from a web page, comprising instructions stored thereon, that when executed by a processor on a client device, perform operations comprising:
requesting and retrieving, by transmitting requests and receiving replies, respectively, at least one web page from a web server;
extracting structured data from the at least one web page;
based on the structured data, determining one or more requests, of the requests, contain parameters for displaying information;
based on a determination that the one or more requests contain the parameters for displaying the information, performing at least one of an extraction operation or a transform operation to obtain the information; and
generating and transmitting at least one request to obtain other information based on the one or more requests.
10. The non-transitory computer-readable medium of claim 9 , further comprising additional instructions stored thereon, that when executed by the processor, perform further operations including:
mapping the other information into a consistent format.
11. The non-transitory computer-readable medium of claim 9 , wherein performing at least one of the extraction operation or the transform operation to obtain the information comprises performing at least one task selected from a group consisting of:
parsing structured information in script tags;
extracting content from received HTML through DOM parsing; or
requesting the content returned in a readable structure.
12. The non-transitory computer-readable medium of claim 9 , wherein extracting the structured data from the at least one web page comprises:
analyzing script tag data within script tags;
generating an abstract syntax tree based on the script tag data; and
traversing and interpreting the abstract syntax tree to extract the structured data.
13. The non-transitory computer-readable medium of claim 9 , wherein extracting the structured data from the at least one web page comprises:
receiving a domain object model for the at least one web page;
analyzing the domain object model; and
based on an analysis result for the domain object model, locating microdata within the domain object model.
14. The non-transitory computer-readable medium of claim 13 , wherein analyzing the domain object model comprises:
applying a model trained by machine learning, to automatically analyze the domain object model.
15. The non-transitory computer-readable medium of claim 9 , wherein extracting the structured data from the at least one web page comprises:
applying a model trained by machine learning, to identify the structured data to be extracted.
16. The non-transitory computer-readable medium of claim 9 , wherein extracting the structured data from the at least one web page comprises:
analyzing a structured format by which the structured data has been organized;
based on an analysis result for the structured format, generating a format for future requests; and
requesting the other information from the web server using the format.
17. A system for extracting data from a web page, comprising:
a processor, on a client device, configured to perform a process including:
requesting and retrieving, by transmitting requests and receiving replies, respectively, at least one web page from a web server;
extracting structured data from the at least one web page;
based on the structured data, determining one or more requests, of the requests, contain parameters for displaying information;
based on a determination that the one or more requests contain the parameters for displaying the information, performing at least one of an extraction operation or a transform operation to obtain the information; and
generating and transmitting at least one request to obtain other information based on the one or more requests.
18. The system of claim 17 , wherein performing at least one of the extraction operation or the transform operation to obtain the information comprises performing at least one task selected from a group consisting of:
parsing structured information in script tags;
extracting content from received HTML through DOM parsing; or
requesting the content returned in a readable structure.
19. The system of claim 17 , wherein extracting the structured data from the at least one web page comprises:
analyzing script tags data within script tags;
generating an abstract syntax tree based on the script tags data; and
traversing and interpreting the abstract syntax tree to extract the structured data.
20. The system of claim 17 , wherein extracting the structured data from the at least one web page comprises:
receiving a domain object model for the at least one web page;
analyzing the domain object model; and
based on an analysis result for the domain object model, locating microdata within the domain object model.