Dynamic optimization of request parameters for proxy server
Systems and methods of task implementation are extended as provided herein and target the web crawling process through a step of submitting a request by a customer to a web crawler. The systems and methods allow a more complex request for a web crawler to be defined in order to receive more specific data. In one aspect, a method for data extraction and gathering from a Network by a Service provider infrastructure include the following steps: checking the parameters of a request received from a User's Device, adjusting the request parameters according to pre-established Scraping logic, selecting a Proxy according to the criteria of the pre-established Scraping logic, sending the adjusted request to the Target through the selected Proxy, checking metadata received from the Target, and forwarding the data to the User's device.
1. A method for data extraction and gathering from a web server comprising:
receiving, by a request optimizer, a request from a user device;
requesting, by the request optimizer, from a proxy rotator a proxy from a proxy database;
accessing, by a proxy rotator, the proxy database;
choosing, by the proxy rotator, from the proxy database the proxy;
providing, by the proxy rotator, to the request optimizer information about the proxy;
sending, by the request optimizer, the request from the user device to the proxy;
obtaining, by the proxy, a reply from a target;
forwarding, by the proxy, the reply obtained from the target to the request optimizer;
analyzing, by the request optimizer, metadata of the reply;
determining, by the request optimizer, pursuant to the analysis, if the reply is positive; and
wherein, if the reply is positive, the reply is sent to the user device;
wherein, if the reply is negative, the method for the data extraction and gathering from the web server is repeated, wherein the repeating comprises deciding, pursuant to the analysis of the reply, if an adjustment of the request is needed, and if the adjustment is needed, the request is modified to be a modified request, and the method for the data extraction and gathering from the web server repeats with the modified request in place of the request.
2. The method of claim 1 , wherein the decision to adjust the request is based on the analysis of the reply from the target, comprising the following:
URL address;
HTTP headers;
Cookies;
Response body.
3. The method of claim 1 , wherein the analysis of the reply includes, but is not limited to:
analyzing details of Hypertext Transfer Protocol (HTTP) reply status;
analyzing the data of the reply.
4. The method of claim 3 , wherein the HTTP reply status may include an HTTP response code:
2xx—Successful responses (positive response);
3xx—Redirects (positive response);
4xx—Client errors (positive response with the exception of 429 , 407 , 408 , 444 , 499 );
5xx—Server errors (negative response).
5. The method of claim 4 , wherein the request parameters may declare specific HTTP response codes as positive or negative for a particular request or multiple requests.
6. The method of claim 5 , wherein the request optimizer identifies the responses with the status codes declared as positive and passes the fetched data to the next data processing step.
7. The method of claim 5 , wherein if the request optimizer identifies responses with status codes declared as negative, the request optimizer adjusts at least one request parameter and repeats the data extraction and gathering from the target.
8. The method of claim 1 , wherein if reply is negative the request is adjusted based on deducing, pursuant to the analysis of the reply, the request parameters and values that are deemed relevant for obtaining a positive reply.
9. The method of claim 1 , wherein the request can be adjusted by:
changing at least one parameter of the request according to directions within a selected scraping strategy;
using another pre-establish scraping strategy;
retrying the scraping request to the target without any changes in the scraping strategy.
10. The method of claim 1 , wherein the proxy transfers the data received from the target without modifying the data received and without checking scope or quality of the data.
11. The method of claim 1 , wherein for communicating the request among the user, the Service Provider Infrastructure and the proxy, an HTTP method is used.
12. The method of claim 1 , wherein more than one proxy server can be manageable by the Service Provider Infrastructure.
13. The method of claim 1 , wherein the proxy can be reused for multiple requests.
14. The method of claim 13 , wherein at least one parameter is set to: session_uuid, session_expires_at, session_remaining.
15. The method of claim 1 , wherein the analysis of the reply leads to creation of different scraping strategies developed by analyzing target parameters and identifying best practices for scraping particular resources.
16. The method of claim 15 wherein scraping strategies are created by:
constructing initial scraping session context, for a particular strategy;
using an initial scraping session to reach the target;
analyzing the data received from the target;
creating a scraping strategy populated with the parameters and their values from the original request combined with the adjustments according to the analysis of the reply.
17. The method of claim 16 , wherein to create a scraping strategy, the initial scraping session has at least one of:
geolocation of the proxy;
timeout threshold for rendering a page in case a headless browser is used;
content elements that must be present to consider the page rendered in case a headless browser is used;
instruction whether to follow redirects;
HTTP status codes that should be considered blocked and retried accordingly;
HTTP status codes that should be considered blocked by captcha and retried accordingly;
HTTP request headers that should always be inserted into the request, regardless if the user included the headers originally;
HTTP request cookies that should always be inserted into the request, regardless if the user included the cookies originally;
HTTP request headers that must be replaced with counterparts that are aligned with the selected scraping strategy;
HTTP request headers that must be removed from the request, even if the user included the headers in the request;
types of HTTP request headers that should be present, for example, ‘User-Agent’ for Chrome or Firefox;
type of proxies that should be used, for example, datacenter, residential, mobile.
18. The method of claim 1 , wherein the reply is positive if the reply is determined to contain proper data or if the reply is determined to not contain an error or a block.