Script-driven data extraction using a browser
View Patent ↗Various example embodiments are directed to a script-driven browsing application for automatically interacting with remote servers. Retrieval of data by virtue of a script-driven browsing application may be facilitated by pairing a script or scripts with a suitable browser, and employing XPath expressions operating on a DOM representation of one or more web pages such that the script-driven browsing application can iterate efficiently and comprehensively over the web pages.
1. A computer-implemented system for automating a browser, the system comprising:
a processor and an operatively-associated memory that stores a script and an argument, wherein the processor executes a routine defined by the script comprising:
initiating a browsing session with a website;
identifying, by using an expression, a parent node within a hierarchical structure representative of at least a portion of the website, wherein the parent node corresponds to an actuatable search element on a graphical user interface;
executing a first operation in association with the parent node, wherein an operand of the first operation is the argument;
identifying, as a result of the first operation, a first child node and a second child node, both the first child node and the second child node on a first web page of a website;
assigning a first context identifier for the first child node;
assigning a second context identifier for the second child node
designating the first child node as a context node from which a loop can be entered;
determining a first document identifier for the first web page;
determining a second document identifier for a second web page, wherein the second web page has a descendant node that is a child of the first child node;
executing a loop operation originating from the first context node, the loop operation extracting information from at least one node on the second web page;
computing a current document identifier during the course of the loop operation; and
in response to determining that the loop operation extracting information from the at least one node on the second web page has ended and to determining that the first document identifier and the second document identifier match, returning control to the first context node and identifying a second context node on the first web page corresponding to a second actuatable search element, wherein the quantity of loops of the loop operation is based on the quantity of linkable elements on the first web page.
2. The computer-implemented system of claim 1 , wherein the first operation in association with the parent node comprises loading one or more associated web pages that result from passing the argument to the website.