Dynamic optimization of request parameters for proxy server
Systems and methods of task implementation are extended as provided herein and target the web crawling process through a step of submitting a request by a customer to a web crawler. The systems and methods allow a more complex request for a web crawler to be defined in order to receive more specific data. In one aspect, a method for data extraction and gathering from a Network by a Service provider infrastructure include the following steps: checking the parameters of a request received from a User's Device, adjusting the request parameters according to pre-established Scraping logic, selecting a Proxy according to the criteria of the pre-established Scraping logic, sending the adjusted request to the Target through the selected Proxy, checking metadata received from the Target, and forwarding the data to the User's device.
1. A method for selecting a scraping strategy, by a request optimizer from a web server, comprising:
initializing a scraping session pursuant to receiving a scraping request from a user device;
examining components of the scraping request;
identifying, within the components of the scraping request, elements relevant for establishing a scraping session context;
establishing scraping session context parameters derived from the elements;
selecting the scraping strategy;
executing the scraping request against the web server, pursuant to applying the scraping strategy selected to the scraping request; and
closing the scraping session after a reply from the web server has been obtained;
wherein application of the scraping strategy to the scraping request results in adding a cookie;
wherein the cookie added to the scraping request results in one of or a combination of:
triggering a particular type of response from a target;
bypassing cookie policy consent pop-up;
bypassing geographical location inquiry pop-up;
bypassing email newsletter popups.
2. The method of claim 1 , wherein the components of the scraping request include at least one of the following:
URL address;
headers;
cookies;
request body.
3. The method of claim 2 , wherein at least one of the components of the scraping request includes custom elements submitted by the user device.
4. The method of claim 2 , wherein, with an exception of URL, none of the components of the scraping request contains custom elements submitted by the user device.
5. The method of claim 1 , wherein the elements relevant for establishing the scraping session context parameters are identified according to pre-established rules of strategy selection.
6. The method of claim 1 , wherein the scraping session context parameters are derived, according to pre-established rules of strategy selection, from the elements identified.
7. The method of claim 1 , wherein the scraping strategy is selected pursuant to comparing the scraping session context parameters to pre-established rules of strategy selection.
8. The method of claim 1 , wherein application of the scraping strategy to the scraping request results in at least one of the following modifications to the scraping request, or a combination thereof:
removing elements of a URL;
adding elements to the URL or an other URL;
modifying elements within the URL or the other URL;
removing the cookie or an other cookie;
adding a header;
removing a header or an other header;
modifying the header or the other header.
9. A method for selecting a scraping strategy, by a request optimizer from a web server, comprising:
initializing a scraping session pursuant to receiving a scraping request from a user device;
examining components of the scraping request;
identifying, within the components of the scraping request, elements relevant for establishing a scraping session context;
establishing scraping session context parameters derived from the elements;
selecting the scraping strategy;
executing the scraping request against the web server, pursuant to applying the scraping strategy selected to the scraping request; and
closing the scraping session after a reply from the web server has been obtained;
wherein application of the scraping strategy to the scraping request results in modifying elements within a URL;
wherein the modifying elements within a URL modification due to the applied scraping strategy results in one of or a combination of:
URL parameter relaying a preference in response presentation;
URL parameter to manipulate scraping request operational flow.
10. The method of claim 9 , wherein the components of the scraping request include at least one of the following:
URL address;
headers;
cookies;
request body.
11. The method of claim 10 , wherein at least one of the components of the scraping request includes custom elements submitted by the user device.
12. The method of claim 10 , wherein, with an exception of URL, none of the components of the scraping request contain custom elements submitted by the user device.
13. The method of claim 9 , wherein the elements relevant for establishing the scraping session context parameters are identified according to pre-established rules of strategy selection.
14. The method of claim 9 , wherein the scraping session context parameters are derived, according to pre-established rules of strategy selection, from the elements identified.
15. The method of claim 9 , wherein the scraping strategy is selected pursuant to comparing the scraping session context parameters to pre-established rules of strategy selection.
16. The method of claim 9 , wherein the cookie added to the scraping request due to the applied scraping strategy results in one of or a combination of:
triggering a particular type of response from a target;
bypassing cookie policy consent pop-up;
bypassing geographical location inquiry pop-up;
bypassing email newsletter popups.
17. A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform the selection of a scraping strategy, by a request optimizer from a web server, comprising:
initializing a scraping session pursuant to receiving a scraping request from a user device;
examining components of the scraping request;
identifying, within the components of the scraping request, elements relevant for establishing a scraping session context;
establishing scraping session context parameters derived from the elements;
selecting the scraping strategy;
executing the scraping request against the web server, pursuant to applying the scraping strategy selected to the scraping request; and
finalizing the scraping session after a reply from the web server has been obtained;
wherein application of the scraping strategy to the scraping request results in adding a cookie;
wherein the cookie added to the scraping request results in one of or a combination of:
triggering a particular type of response from a target;
bypassing cookie policy consent pop-up;
bypassing geographical location inquiry pop-up;
bypassing email newsletter popups.
18. The non-transitory computer-readable device of claim 17 , wherein the components of the scraping request include at least one of the following:
URL address;
headers;
cookies;
request body.
19. The non-transitory computer-readable device of claim 18 , wherein at least one of the components of the scraping request includes custom elements submitted by the user device.
20. The non-transitory computer-readable device of claim 18 , wherein with an exception of URL, none of the components of the scraping request contains custom elements submitted by the user device.
21. The non-transitory computer-readable device of claim 17 , wherein the elements relevant for establishing the scraping session context parameters are identified according to pre-established rules of strategy selection.
22. A system for selecting a scraping strategy comprising:
at least one processor;
a memory coupled to the at least one processor;
a request optimizer configured to:
initialize a scraping session pursuant to receiving a scraping request from a user device;
examine components of the scraping request;
identify, within the components of the scraping request, elements relevant for establishing a scraping session context;
establish scraping session context parameters derived from the elements;
select the scraping strategy;
execute the scraping request against the web server, pursuant to applying the scraping strategy selected to the scraping request; and
finalize the scraping session after a reply from the web server has been obtained;
wherein application of the scraping strategy to the scraping request results in adding a cookie;
wherein the cookie added to the scraping request results in one of or a combination of:
triggering a particular type of response from a target;
bypassing cookie policy consent pop-up;
bypassing geographical location inquiry pop-up;
bypassing email newsletter popups.
23. The system of claim 22 , wherein the scraping session context parameters are derived, according to pre-established rules of strategy selection, from the elements identified.
24. The system of claim 22 , wherein the strategy is selected pursuant to comparing the scraping session context parameters to pre-established rules of strategy selection.
25. The system of claim 22 , wherein application of the scraping strategy to the scraping request results in at least one of the following modifications to the scraping request, or a combination thereof:
removing elements of a URL;
adding elements to the URL or an other URL;
modifying elements within the URL or the other URL;
removing the cookie or an other cookie;
adding a header;
removing a header or an other header;
modifying the header or the other header.