Automated world wide web navigation and content extraction
Storage mediums and a computer-implemented method for automating web navigation and content extraction are provided. In particular, a storage medium with program components which are executable through a common application program interface and are utilizable by a developer to write programming instructions is provided. In some cases, the storage medium may include a program component for adaptively navigating through one or more websites and another program component for extracting scripted content from the one or more websites. In addition or alternatively, the storage medium may include a program component for standardizing content on a web page. In some cases, the storage medium may be configured to allow a user to include XPath query language in program instructions written from the storage medium. A storage medium comprising program instructions executable using a processor for performing such functions and a computer-implemented method employing such processes are also provided herein.
1. A storage medium comprising a Software Development Kit (SDK) having program components which are executable through a common application program interface, wherein the program components comprise:
a first program component having coding directives which are utilizable by a developer to write program instructions that are executable by a processor for adaptively navigating through one or more websites,
wherein adaptively navigating includes adjusting parameters used to dynamically avoid or overcome obstacles encountered along a navigational route, where the parameters adjusted includes at least one of the following parameters—changing browser identity, executing scripted content on web pages or manipulating form controls;
one or more additional program components having coding directives which are utilizable by a developer to write program instructions that are executable by a processor for:
extracting scripted content, wherein at least some of the content is available, but not displayed, from the one or more websites including loading and executing the scripted content; and
storing the extracted scripted content at a target location.
2. The storage medium of claim 1 , wherein the coding directives of the first program component are utilizable by a developer to write program instructions that are executable by a processor for conditionally navigating through the one or more websites.
3. The storage medium of claim 1 , wherein the coding directives of the first program component are utilizable by a developer to write program instructions that are executable by a processor for facilitating navigation through the one or more websites.
4. The storage medium of claim 3 , wherein the coding directives of the first program component are utilizable by the developer to selectively write the program instructions associated with facilitated navigation for specific timeframes.
5. The storage medium of claim 1 , wherein the coding directives of the one or more additional program components are further utilizable by a developer to write program instructions that are executable by a processor for standardizing the scripted and unscripted content.
6. The storage medium of claim 1 , wherein the coding directives of the one or more additional program components are further utilizable by a developer to write program instructions that are executable by a processor for generating a model of logical structure of the scripted and unscripted content.
7. The storage medium of claim 1 , wherein the coding directives of the first program component are further utilizable by a developer to write program instructions that are executable by a processor for:
recognizing a scripting language embedded within the one or more websites; and
executing the embedded scripting language using an interpreter.
8. The storage medium of claim 1 , wherein the coding directives of the first program component are utilizable by a developer to write program instructions that are executable by a processor for accessing data other than what may be configured to be displayed on a browser as characterized by a structural layout of an accessed website.
9. The storage medium of claim 1 , wherein the coding directives of the one or more program components are further utilizable by a developer to write program instructions that are executable by a processor for posting data on the one or more websites.
10. The storage medium of claim 1 , wherein the coding directives of the first program component and one or more additional program components are utilizable by a developer to write event driven program instructions.
11. A Software Development Kit (SDK) comprising program instructions executable using a processor for:
adaptively navigating through one or more websites to access information,
wherein adaptively navigating includes adjusting parameters used to dynamically avoid or overcome obstacles encountered along a navigational route, where the parameter adjusted includes at least one of the following parameters—changing browser identity, executing scripted content on web pages or manipulating form controls;
parsing the accessed information into a model of logical structure;
loading and executing a scripting language embedded within the one or more websites such that results of the script execution can be parsed into the model of logical structure;
searching for content, at least some of which is available but not displayed on said one or more websites within the model of logical structure;
extracting, independent of user intervention, the searched content from the one or more websites; and
storing, independent of user intervention, the extracted content at a target location.
12. The storage medium of claim 11 , wherein the program instructions are further for accessing the website without a user interface.
13. The storage medium of claim 11 , wherein the program instructions are further for posting data upon the website.
14. The storage medium of claim 11 , wherein the program instructions are for monitoring the status of the accessed information on the website, and for sending an alert upon detecting a change in the status of the accessed information.
15. A computer-implemented method for obtaining a collection of information from one or more websites, comprising:
accessing the one or more websites;
partitioning contents on the one or more websites into a model of logical structure;
loading and executing a script embedded within the one or more websites such that results of the script execution can be parsed into the model of logical structure;
querying the model of logical structure for information of interest;
automatically extracting, independent of user intervention, the information of interest from the one or more websites;
automatically storing, independent of user intervention, the extracted information of interest at a target location; and
adaptively navigating one or more websites based, in part, on the extracted information of interest;
wherein adaptively navigating includes adjusting parameters used to dynamically avoid or overcome obstacles encountered along a navigational route where the adjusted parameter includes at least one of the following parameters—changing browser identity, executing scripted content on web pages or manipulating form controls.
16. The computer-implemented method of claim 15 , further comprising standardizing the contents on the one or more websites into a standard format prior to the step of partitioning.
17. The computer-implemented method of 15 , further comprising posting data upon a website in response to the step of extracting the information of interest from the one or more websites.
18. The computer-implemented method of claim 15 , including monitoring the status of the contents on the one or more websites, and further comprising performing the steps of partitioning, querying, automatically extracting, and automatically storing upon detecting a change in the status of the contents on the one or more websites.