Automated extraction of data from web pages
Various embodiments provide techniques for automatically extracting data from web pages. Such extraction can take place without the use of a browser, and without necessarily rendering the entire web page. Thus, data extraction can be performed more efficiently and more quickly, while reducing the computing resources needed to perform such operations. In at least one embodiment, data extraction and translation are performed by automatically parsing structured data from visible and hidden elements of a web page.
1. A computer-implemented method for extracting data from a web page, comprising:
at a processor on a client device, requesting and retrieving, by transmitting requests and receiving replies, respectively, at least one web page from a web server;
at the processor, automatically extracting structured data from the at least one web page;
at the processor, based on the extracted structured data, automatically determining one or more requests, of the requests, contain parameters for displaying information;
at the processor, based on the determination, automatically performing at least one of an extraction operation or a transform operation to obtain the information;
at the processor, automatically generating at least one request based on the determined one or more requests to obtain other information;
at the processor, automatically transmitting the generated at least one request;
at the processor, receiving at least one response to the transmitted at least one request; and
at the processor, automatically parsing the received at least one response to extract the other information by performing the at least one of the extraction operation or the transform operation on the at least one response.
2. The computer-implemented method of claim 1 , wherein the receiving and the parsing the at least one response are performed without rendering the response for display.
3. The computer-implemented method of claim 1 , further comprising:
automatically storing the extracted other information at a storage device.
4. The computer-implemented method of claim 1 , further comprising:
automatically outputting the extracted other information at an output device.
5. The computer-implemented method of claim 1 , further comprising:
automatically transmitting the extracted other information via an electronic network.
6. The computer-implemented method of claim 1 , further comprising:
automatically mapping the extracted other information into a consistent format.
7. The computer-implemented method of claim 1 , wherein the automatically performing the at least one of the extraction operation or the transform operation to obtain the information comprises automatically performing at least one selected from a group consisting of:
automatically parsing structured information in script tags;
automatically extracting content from received HTML through DOM parsing; or
automatically requesting content returned in a readable structure.
8. The computer-implemented method of claim 1 , wherein the extracted other information comprises product data.
9. The computer-implemented method of claim 1 , wherein the extracted other information comprises data describing product variants.
10. The computer-implemented method of claim 1 , wherein the automatically extracting the structured data from the at least one web page comprises:
automatically analyzing data within script tags;
automatically generating an abstract syntax tree based on the analyzed data; and
automatically traversing and interpreting the abstract syntax tree to extract the structured data.
11. The computer-implemented method of claim 1 , wherein the automatically extracting the structured data from the at least one web page comprises:
receiving a domain object model for the at least one web page;
automatically analyzing the domain object model; and
based on the analysis of the domain object model, automatically locating microdata within the domain object model.
12. The computer-implemented method of claim 11 , wherein the automatically analyzing the domain object model comprises:
applying a model trained by machine learning, to automatically analyze the domain object model.
13. The computer-implemented method of claim 1 , wherein the automatically extracting the structured data from the at least one web page comprises:
applying a model trained by machine learning, to identify the structured data to be extracted.
14. The computer-implemented method of claim 1 , wherein the automatically extracting the structured data from the at least one web page comprises:
analyzing a structured format by which the data has been organized;
based on the analysis of the structured format, generating a format for future requests; and
requesting information from the web server using the generated format.
15. The computer-implemented method of claim 1 , wherein the web page comprises an e-commerce web page, and wherein the extracted other information describes an item available via the e-commerce web page.
16. The computer-implemented method of claim 1 , wherein the information obtained by the at least one of the extraction operation or the transform operation comprises domain-specific information.
17. A non-transitory computer-readable medium for extracting data from a web page, comprising instructions stored thereon, that when executed by a processor on a client device, perform the steps of:
requesting and retrieving, by transmitting requests and receiving replies, respectively, at least one web page from a web server;
automatically extracting structured data from the at least one web page;
based on the extracted structured data, automatically determining one or more requests, of the requests, contain parameters for displaying information;
based on the determination, automatically performing at least one of an extraction operation or a transform operation to obtain the information;
automatically generating at least one request to obtain other information based on the determined one or more requests;
automatically transmitting the generated at least one request;
receiving at least one response to the transmitted at least one request; and
automatically parsing the received at least one response to extract the other information by performing the at least one of the extraction operation or the transform operation on the at least one response.
18. The non-transitory computer-readable medium of claim 17 , wherein the receiving and the parsing the at least one response are performed without rendering the response for display.
19. The non-transitory computer-readable medium of claim 17 , further comprising instructions stored thereon, that when executed by the processor, perform the step of:
causing a storage device to automatically store the extracted other information.
20. The non-transitory computer-readable medium of claim 17 , further comprising instructions stored thereon, that when executed by the processor, perform the step of:
causing an output device to automatically output the extracted other information.
21. The non-transitory computer-readable medium of claim 17 , further comprising instructions stored thereon, that when executed by the processor, perform the step of:
automatically transmitting the extracted other information via an electronic network.
22. The non-transitory computer-readable medium of claim 17 , further comprising instructions stored thereon, that when executed by the processor, perform the step of:
automatically mapping the extracted other information into a consistent format.
23. The non-transitory computer-readable medium of claim 17 , wherein the automatically performing the at least one of the extraction operation or the transform operation to obtain the information comprises automatically performing at least one selected from a group consisting of:
automatically parsing structured information in script tags;
automatically extracting content from received HTML through DOM parsing; or
automatically requesting content returned in a readable structure.
24. The non-transitory computer-readable medium of claim 17 , wherein the extracted other information comprises product data.
25. The non-transitory computer-readable medium of claim 17 , wherein the extracted other information comprises data describing product variants.
26. The non-transitory computer-readable medium of claim 17 , wherein the automatically extracting the structured data from the at least one web page comprises:
automatically analyzing data within script tags;
automatically generating an abstract syntax tree based on the analyzed data; and
automatically traversing and interpreting the abstract syntax tree to extract the structured data.
27. The non-transitory computer-readable medium of claim 17 , wherein the automatically extracting the structured data from the at least one web page comprises:
receiving a domain object model for the at least one web page;
automatically analyzing the domain object model; and
based on the analysis of the domain object model, automatically locating microdata within the domain object model.
28. The non-transitory computer-readable medium of claim 27 , wherein the automatically analyzing the domain object model comprises:
applying a model trained by machine learning, to automatically analyze the domain object model.
29. The non-transitory computer-readable medium of claim 17 , wherein the automatically extracting the structured data from the at least one web page comprises:
applying a model trained by machine learning, to identify the structured data to be extracted.
30. The non-transitory computer-readable medium of claim 17 , wherein the automatically extracting the structured data from the at least one web page comprises:
analyzing a structured format by which the data has been organized;
based on the analysis of the structured format, generating a format for future requests; and
requesting information from the web server using the generated format.
31. The non-transitory computer-readable medium of claim 17 , wherein the web page comprises an e-commerce web page, and wherein the extracted other information describes an item available via the e-commerce web page.
32. The non-transitory computer-readable medium of claim 17 , wherein the information obtained by the at least one of the extraction operation or the transform operation comprises domain-specific information.
33. A system for extracting data from a web page, comprising:
a processor, on a client device, configured to perform a process including:
requesting and retrieving, by transmitting requests and receiving replies, respectively, at least one web page from a web server;
automatically extracting structured data from the at least one web page;
based on the extracted structured data, automatically determining one or more requests, of the requests, contain parameters for displaying information;
based on the determination, automatically performing at least one of an extraction operation or a transform operation to obtain the information;
automatically generating at least one request to obtain other information based on the determined one or more requests;
automatically transmitting the generated at least one request;
receiving at least one response to the transmitted at least one request; and
automatically parsing the received at least one response to extract the other information by performing the at least one of the extraction operation or the transform operation on the at least one response.
34. The system of claim 33 , wherein the receiving and the parsing the at least one response are performed without rendering the response for display.
35. The system of claim 33 , further comprising:
a storage device, communicatively coupled to the processor, configured to automatically store the extracted other information.
36. The system of claim 33 , further comprising:
an output device, communicatively coupled to the processor, configured to automatically output the extracted other information.
37. The system of claim 33 , further comprising:
a transmission component, communicatively coupled to the processor, configured to automatically transmit the extracted other information via an electronic network.
38. The system of claim 33 , wherein the processor is further configured to automatically map the extracted other information into a consistent format.
39. The system of claim 33 , wherein the automatically performing the at least one of extraction operation or transform operation to obtain the information comprises automatically performing at least one selected from a group consisting of:
automatically parsing structured information in script tags;
automatically extracting content from received HTML through DOM parsing; or
automatically requesting content returned in a readable structure.
40. The system of claim 33 , wherein the extracted other information comprises product data.
41. The system of claim 33 , wherein the extracted other information comprises data describing product variants.
42. The system of claim 33 , wherein the automatically extracting the structured data from the at least one web page comprises:
automatically analyzing data within script tags;
automatically generating an abstract syntax tree based on the analyzed data; and
automatically traversing and interpreting the abstract syntax tree to extract the structured data.
43. The system of claim 33 , wherein the automatically extracting the structured data from the at least one web page comprises:
receiving a domain object model for the at least one web page;
automatically analyzing the domain object model; and
based on the analysis of the domain object model, automatically locating microdata within the domain object model.
44. The system of claim 43 , wherein the automatically analyzing the domain object model comprises:
applying a model trained by machine learning, to automatically analyze the domain object model.
45. The system of claim 33 , wherein the automatically extracting the structured data from the at least one web page comprises:
applying a model trained by machine learning, to identify the structured data to be extracted.
46. The system of claim 33 , wherein the automatically extracting the structured data from the at least one web page comprises:
analyzing a structured format by which the data has been organized;
based on the analysis of the structured format, generating a format for future requests; and
requesting information from the web server using the generated format.
47. The system of claim 33 , wherein the web page comprises an e-commerce web page, and wherein the extracted other information describes an item available via the e-commerce web page.
48. The system of claim 33 , wherein the information comprises domain-specific information.