Smart proxy rotator
The task, logic of HTTP/HTTPS session statistics interception and collection is moved to the client side instead of the proxy layer. Encrypted HTTPS tunnel is terminated at the client end, making the actual content or data in transit invisible to both proxies and the smart proxy rotator (SPR). Client's scraping software has a plug-in installed that expands its functionality. HTTP/HTTPS session quality metrics are intercepted and collected at the client side, then sent to the SPR. Proxy usage mark “can be used” is obtained from the SPR for the currently analyzed proxy, based on the results of metrics analysis.
1. A computer-implemented method for dynamically configuring a utilization threshold of a proxy provider, the method comprising:
running a smart proxy rotator (SPR) on a computing device connected to a network;
obtaining a proxy from the SPR, according to currently valid SPR operational settings, by a Web Scraper to use for a request for content;
establishing, by the Web Scraper, an encrypted connection to a target server through the proxy;
requesting and obtaining the content designated within the request, through the encrypted connection, by the Web Scraper;
providing, by the Web Scraper, performance and availability information of the proxy to the SPR;
calculating, at the SPR, a weight for the proxy provider; and
dynamically configuring utilization threshold of the proxy provider based on the calculation.
2. The method of claim 1 , wherein the Web Scraper accumulates the performance and availability information from a plurality of proxies before providing the information to the SPR.
3. The method of claim 1 , wherein proxy providers' weights are calculated by the SPR based on the evaluation of performance and availability information from at least one proxy belonging to the proxy provider.
4. The method of claim 1 further comprising dynamically configuring utilization threshold of the proxy provider based on the weight calculations.
5. The method of claim 1 , wherein the SPR periodically calculates and configures the proxy provider weight according to the proxy performance and availability information submitted by the Web Scraper at any configured period of time.
6. The method of claim 1 , wherein the SPR stops using a maximized proxy provider, at least for a period of time, when at least one configured maximum threshold is reached.
7. The method of claim 1 , wherein the performance and availability information collected include, but are not limited to: proxy id, proxy status, proxy response time, and a size of data obtained through the proxy.
8. The method of claim 1 , wherein SPR operational settings, such as proxy provider settings, utilization thresholds, error thresholds and proxy lists, are loaded into a database on disk, a flat file on disk, a database in memory, or other storage media.
9. The method of claim 1 , wherein the SPR can be placed within a client's network, proxy service provider's network, or a party providing proxy rotation services.
10. A non-transitory computer readable medium computer medium for dynamically configuring a utilization threshold of a proxy provider comprising instructions which, when executed by a computing device, causes the computing device to:
provide a Web Scraper with a proxy from a proxy provider list;
accumulate performance and availability information about the proxy provided by the Web Scraper after the proxy has been used to obtain content from a target, wherein the content is unreadable to a Smart Proxy Rotator (SPR) due to network transport layer or application layer encryption;
calculate and configure utilization thresholds of a plurality of proxy providers by evaluating proxy performance and availability information received from the Web Scraper.
11. The non-transitory computer-readable medium of claim 10 , wherein the proxy provider's utilization threshold is periodically reconfigured according to the performance and availability information provided by the Web Scraper.
12. The non-transitory computer-readable medium of claim 10 , wherein proxy providers' weights are periodically calculated and configured according to the proxy performance and availability information submitted by the Web Scraper at any configured period of time.
13. The non-transitory computer-readable medium of claim 10 , wherein usage of the proxy provider is placed on hold, at least for a period of time, when at least one configured maximum threshold is reached.
14. The non-transitory computer-readable medium of claim 10 , wherein the performance and availability information collected include, but are not limited to: proxy id, proxy status, proxy response time, and a size of data obtained through the proxy.
15. The non-transitory computer-readable medium of claim 10 , wherein operational settings, such as proxy provider settings, utilization thresholds, error thresholds and proxy lists, are loaded into a database on disk, a flat file on disk, a database in memory, or other storage media.
16. The non-transitory computer-readable medium of claim 10 , wherein the computing device executing the program can be placed within a client's network, proxy service provider's network, or any other network.
17. A system comprising:
a Smart Proxy Rotator (SPR) operable to provide a plurality of proxies over a network; and
a Web Scraper operable to obtain a first proxy of the plurality of proxies from the SPR according to currently valid SPR operational settings, to establish an encrypted connection with the target server through the first proxy, to request content from a target server, to receive the content designated within the request through the encrypted connection, and to provide performance and availability information of the first proxy to the SPR,
wherein the SPR is operable to calculate a weight for the proxy provider of the first proxy based on first proxy performance statistics analysis, and to dynamically configure utilization threshold of the proxy provider based on the weight calculated.
18. The system of claim 17 , wherein proxy providers' weights are calculated by the SPR based on the evaluation of performance and availability information from at least one proxy belonging to the provider.
19. The system of claim 17 , wherein the SPR periodically, at any configured period of time, calculates and configures proxy providers' weights according to the proxy performance and availability information submitted by the Web Scraper.
20. The system of claim 17 , wherein the SPR stops using a maximized proxy provider, at least for a period of time, when at least one configured maximum threshold is reached.