Automated sequential site navigation
A website is navigated automatically in a sequential manner, for example by performing a series of specific network requests in a specified order, without necessarily using a browser, and without necessarily rendering each web page in the sequence. Interactions with a website are automatically recorded, stored, and analyzed. Requests and responses can be associated with particular in-put/output operations. Based on the analysis, the system can determine what data is expected by web servers, so that it can effectively simulate browser operations in connection with those web servers, and extract useful information from the data received in response to those operations. Thus, data extraction and/or other operations can be performed more efficiently and more quickly, while reducing the computing resources needed.
1. A computer-implemented method on a client device for automatically navigating a website, comprising:
obtaining a plurality of interactions and a plurality of network requests, the plurality of network requests corresponding to one or more of the plurality of interactions that take place during performance of a web-based operation including communications with a web server, the plurality of interactions inputting data, the plurality of network requests including encoded data corresponding to the data;
analyzing the plurality of interactions and the plurality of network requests to determine an encoding scheme of the web server by:
determining a request, of the plurality of network requests, contains a first data of the data of the plurality of interactions, and
swapping the first data in the request for a key to be used for future requests;
generating a rewritten network request based on the encoding scheme, the rewritten network request including substitute data in place of the encoded data by inserting the substitute data for the key in the request; and
executing the rewritten network request.
2. The computer-implemented method of claim 1 , wherein the executing the rewritten network request includes transmitting the rewritten network request to the web server.
3. The computer-implemented method of claim 1 , further comprising, prior to generating the rewritten network request, identifying network requests of the plurality of network requests necessary to complete the web based operation.
4. The computer-implemented method of claim 1 , wherein the web-based operation includes displaying a plurality of web pages.
5. The computer-implemented method of claim 1 , wherein the web-based operation includes a web-based purchase.
6. The computer-implemented method of claim 1 , wherein each network request includes a communication between the client device and the web server.
7. The computer-implemented method of claim 1 , wherein the executing the rewritten network request includes automatically executing the rewritten network request without using a browser.
8. The computer-implemented method of claim 1 , wherein the executing the rewritten network request includes automatically executing the rewritten network request without rendering results for display.
9. The computer-implemented method of claim 1 , wherein the executing the rewritten network request includes automatically executing the rewritten network request without human interaction.
10. The computer-implemented method of claim 1 , further comprising extracting information from the website while executing the rewritten network request.
11. A non-transitory computer-readable medium storing instructions that, when executed by a processor on a client device, cause the processor to perform a method for automatically navigating a website, the method comprising:
obtaining a plurality of interactions and a plurality of network requests, the plurality of network requests corresponding to one or more of the plurality of interactions that take place during performance of a web-based operation including communications with a web server, the plurality of interactions inputting data, the plurality of network requests including encoded data corresponding to the data;
analyzing the plurality of interactions and the plurality of network requests to determine an encoding scheme of the web server by:
determining a request, of the plurality of network requests, contains a first data of the data of the plurality of interactions, and
swapping the first data in the request for a key to be used for future requests;
generating a rewritten network request based on the encoding scheme, the rewritten network request including substitute data in place of the encoded data by inserting the substitute data for the key in the request; and
executing the rewritten network request.
12. The non-transitory computer-readable medium of claim 11 , wherein the method further comprises, prior to the generating the rewritten network request, automatically identifying network requests of the plurality of network requests necessary to complete the web-based operation.
13. The non-transitory computer-readable medium of claim 11 , wherein the web-based operation includes displaying a plurality of web pages.
14. The non-transitory computer-readable medium of claim 11 , wherein the web-based operation includes a web-based purchase.
15. The non-transitory computer-readable medium of claim 11 , wherein each network request includes a communication between the client device and the web server.
16. The non-transitory computer-readable medium of claim 11 , wherein the executing the rewritten network request includes automatically executing the rewritten network request without using a browser.
17. The non-transitory computer-readable medium of claim 11 , wherein the executing the rewritten network request includes automatically executing the rewritten network request without rendering results for display.
18. The non-transitory computer-readable medium of claim 11 , wherein the executing the rewritten network request includes automatically executing the rewritten network request without human interaction.
19. The non-transitory computer-readable medium of claim 11 , wherein the method further comprises automatically extracting information from the website while executing the rewritten network request.
20. A system for automatically navigating a website, comprising:
a memory storing instructions; and
a processor executing the instructions to perform a process, the process including:
obtaining a plurality of interactions and a plurality of network requests, the plurality of network requests corresponding to one or more of the plurality of interactions that take place during performance of a web-based operation including communications with a web server, the plurality of interactions inputting data, the plurality of network requests including encoded data corresponding to the data;
analyzing the plurality of interactions and the plurality of network requests to determine an encoding scheme of the web server by:
determining a request, of the plurality of network requests, contains a first data of the input data of the plurality of interactions, and
swapping the first data in the request for a key to be used for future requests;
generating a rewritten network request based on the encoding scheme, the rewritten network request including substitute data in place of the encoded data by inserting the substitute data for the key in the request; and
executing the rewritten network request.