Dynamic optimization of request parameters for proxy server
Systems and methods of task implementation are extended as provided herein and target the web crawling process through a step of submitting a request by a customer to a web crawler. The systems and methods allow a more complex request for a web crawler to be defined in order to receive more specific data. In one aspect, a method for data extraction and gathering from a Network by a Service provider infrastructure include the following steps: checking the parameters of a request received from a User's Device, adjusting the request parameters according to pre-established Scraping logic, selecting a Proxy according to the criteria of the pre-established Scraping logic, sending the adjusted request to the Target through the selected Proxy, checking metadata received from the Target, and forwarding the data to the User's device.
1. A method for data extraction and gathering from a web server comprising:
comparing parameters of a request received from a user device with a list of scraping strategies;
selecting, from the list of scraping strategies, a scraping strategy corresponding to the parameters of the request;
identifying discrepancies between the request and the scraping strategy;
adjusting the parameters of the request according to the scraping strategy to form an adjusted request;
sending the adjusted request to a target through a proxy selected according to the scraping strategy;
receiving a reply from the target.
2. The method of claim 1 , further comprising:
checking and analyzing the metadata of the reply received from the target.
3. The method of claim 2 , further comprising:
forwarding the reply to the user device pursuant to the analysis of the metadata.
4. The method of claim 1 , wherein if the reply received from the target is negative, the method repeats by adjusting the parameters of the request according to a different scraping strategy from the list of scraping strategies to form a different adjusted request that is then sent to the target.
5. The method of claim 4 , wherein the method is repeated with alternative scraping strategies selected from the list of scraping strategies until a positive response is obtained or until a maximum number of attempts is reached.
6. The method of claim 1 , wherein at least one of the parameters within the following components of the request is adjusted:
a. URL address;
b. headers;
c. cookies;
d. request body.
7. The method of claim 1 , wherein the proxy is selected by a service provider infrastructure from a proxy database.
8. The method of claim 7 , wherein the proxy is selected according to the scraping strategy and includes geolocation settings, proxy type, proxy availability, or any combination thereof.
9. The method of claim 1 , wherein the proxy remains available for multiple user device requests.
10. The method of claim 1 , wherein the parameters of the request are not adjusted if the request received from the user device is aligned to one of the scraping strategies from the list.
11. The method of claim 1 , wherein request optimizer operational settings include at least one of the following: scraping strategies, session ID numbers, servers accepted as request originators, operation timeframes, scheduled maintenance jobs, connectivity sessions timeouts, request processing logs, or any combination thereof and such settings are stored in a database on disk, in a flat file on disk, in a database in memory, or other storage media.
12. The method of claim 1 , wherein the target is a web server serving the reply accessible by HTTP/HTTPS protocols and is attached to a network.
13. The method of claim 1 , wherein the proxy is a proxy server integrated into a service provider infrastructure.
14. The method of claim 1 , wherein the proxy is a proxy server separate from a service provider infrastructure.
15. The method of claim 1 , wherein upon receiving the request, a service provider infrastructure performs the adjusting of the parameters of the request autonomously from the user device.
16. A system for data extraction and gathering from a network comprising at least:
a proxy rotator operable to provide a plurality of proxies over the network; and
a request optimizer operable to compare parameters of a request received from a user device with a list of scraping strategies, to select a scraping strategy from the list of scraping strategies a scraping strategy that corresponds to the parameters of the request, to identify discrepancies between the request and the scraping strategy, to adjust the parameters of the request according to the scraping strategy to form an adjusted request, and to send the adjusted request to a target through a proxy selected from a proxy database by the proxy rotator according to the scraping strategy.
17. The system of claim 16 , wherein the request optimizer can adjust the request received from the user device by modifying at least one of the parameters within the following components:
a. URL address;
b. headers;
c. cookies;
d. request body.
18. The system of claim 16 , wherein the request optimizer utilizes the same proxy for an unlimited number of requests received from the same user device.
19. A non-transitory computer readable medium for data extraction and gathering from a web server comprising instructions which, when executed by a computing device within a service provider infrastructure, cause the computing device to perform:
comparing parameters of a request received from a user device with a list of scraping strategies;
selecting, from the list of scraping strategies, a scraping strategy corresponding to the parameters of the request;
identifying discrepancies between the request and the scraping strategy;
adjusting the parameters of the request according to the scraping strategy to form an adjusted request;
sending the adjusted request to a target through a proxy selected according to the scraping strategy;
receiving a reply from the target.
20. The method of claim 19 , further comprising:
checking and analyzing the metadata of the reply received from the target.
21. The method of claim 20 further comprising:
forwarding the reply to the user device pursuant to the analysis of the metadata.
22. The non-transitory computer readable medium of claim 19 , wherein if the reply received from the target is negative, operation of the medium repeats by adjusting the parameters of the request according to a different scraping strategy from the list of scraping strategies to form a different adjusted request that is then sent to the target.
23. The non-transitory computer readable medium of claim 22 wherein the operation of the medium is repeated with alternative scraping strategies selected from the list of scraping strategies until a positive response is obtained or until a maximum number of attempts is reached.
24. The non-transitory computer readable medium of claim 19 , wherein the proxy remains available for multiple user device requests.
25. The non-transitory computer readable medium of claim 19 , wherein at least one of the following parameters of the request is adjusted:
a. URL address;
b. headers;
c. cookies;
d. Request body.
26. The non-transitory computer readable medium of claim 19 , wherein the parameters of the request are not adjusted if the request received from the user device is aligned to one of the scraping strategies from the list.