Automated sequential site navigation
A website is navigated automatically in a sequential manner, for example by performing a series of specific network requests in a specified order, without necessarily using a browser, and without necessarily rendering each web page in the sequence. Interactions with a website are automatically recorded, stored, and analyzed. Requests and responses can be associated with particular input/output operations. Based on the analysis, the system can determine what data is expected by web servers, so that it can effectively simulate browser operations in connection with those web servers, and extract useful information from the data received in response to those operations. Thus, data extraction and/or other operations can be performed more efficiently and more quickly, while reducing the computing resources needed.
1. A computer-implemented method on a client device for interacting with web based systems, comprising:
obtaining a communication structure of a web based system, based on recording interactions with the web based system, the communication structure comprising:
a request containing a first data, and
a key to manipulate the first data in the request;
generating a series of rewritten network requests based on the communication structure, one or more rewritten network requests of the series of rewritten network requests including substitute data in place of the first data by implementing the substitute data based on the key in the request; and
executing one or more rewritten network requests of the series of rewritten network requests based on the communication structure.
2. The computer-implemented method of claim 1 , wherein the executing one or more rewritten network requests of the series of rewritten network requests includes transmitting at least one rewritten network request to the web based system.
3. The computer-implemented method of claim 1 , further comprising, prior to generating a series of rewritten network requests, identifying network requests of a plurality of network requests necessary to complete a web based operation.
4. The computer-implemented method of claim 3 , wherein the web based operation includes displaying a plurality of web pages.
5. The computer-implemented method of claim 3 , wherein the web based operation includes a web based purchase.
6. The computer-implemented method of claim 3 , wherein each network of the plurality of network requests includes a communication between the client device and the web based system.
7. The computer-implemented method of claim 1 , wherein the executing one or more rewritten network requests of the series of rewritten network requests includes automatically executing one or more rewritten network requests without using a browser.
8. The computer-implemented method of claim 1 , wherein the executing one or more rewritten network requests of the series of rewritten network requests includes automatically executing one or more rewritten network requests without rendering results for display.
9. The computer-implemented method of claim 1 , the executing one or more rewritten network requests of the series of rewritten network requests includes automatically executing one or more rewritten network requests without human interaction.
10. The computer-implemented method of claim 1 , further comprising extracting information from a response of the web based system in response to receiving the response to one or more rewritten network requests of the series of rewritten network requests.
11. A computer-implemented method on a client device for interacting with web based systems, comprising:
obtaining a communication structure of a web based system, based on recording interactions with the web based system, the communication structure comprising:
a request containing a first data, and
a key to manipulate the first data in the request;
identifying a network request from a plurality of network requests, the network request utilized to complete a web based operation at the web based system;
generating a rewritten network request based on the communication structure and the network request, the rewritten network request including first substitute data in place of the first data by implementing the first substitute data based on the key in the request; and
executing the rewritten network request based on the communication structure.
12. The computer-implemented method of claim 11 , wherein recording interactions with the web based system comprises recording requests made during interactions between a browser and the web based system.
13. The computer-implemented method of claim 11 , wherein recording interactions with the web based system comprises recording interactions performed by an automated site navigation module that interacts with the web based system.
14. The computer-implemented method of claim 11 , wherein the executing the rewritten network request includes transmitting the rewritten network request to the web based system.
15. The computer-implemented method of claim 11 , wherein the web based operation includes one or more of displaying a plurality of web pages or a web based purchase.
16. The computer-implemented method of claim 11 , wherein each network of the plurality of network requests includes a communication between the client device and the web based system.
17. The computer-implemented method of claim 11 , wherein the executing the rewritten network request includes automatically executing the rewritten network request without using a browser.
18. The computer-implemented method of claim 11 , wherein the executing the rewritten network request includes automatically executing the rewritten network request without rendering results for display.
19. The computer-implemented method of claim 11 , wherein the executing the rewritten network request includes automatically executing the rewritten network request without human interaction.
20. A system for automatically interacting with web based systems, comprising:
a memory storing instructions; and
a processor executing the instructions to perform a process based on recording interactions with a web based system, the process including:
recording interactions with a web based system;
analyzing the interactions with the web based system to determine a data format;
identifying one or more requests necessary to complete an operation based on the data format; and
obtaining an communication structure of the web based system based on the requests necessary to complete the operation, the communication structure comprising a key to implement first data in a network request for generation of a rewritten network request based on the communication structure and the network request, the rewritten network request including first substitute data in place of the first data by implementing the first substitute data based on the key.