Smart proxy rotator
The task, logic of HTTP/HTTPS session statistics interception and collection is moved to the client side instead of the proxy layer. Encrypted HTTPS tunnel is terminated at the client end, making the actual content or data in transit invisible to both proxies and the smart proxy rotator (SPR). Client's scraping software has a plug-in installed that expands its functionality. HTTP/HTTPS session quality metrics are intercepted and collected at the client side, then sent to the SPR. Proxy usage mark “can be used” is obtained from the SPR for the currently analyzed proxy, based on the results of metrics analysis.
1. A non-transitory computer readable medium for identifying a threshold of at least one proxy provider, the medium comprising instructions that, when executed by a processor, direct the processor to:
provide a web scraper with a first proxy from a first proxy provider;
establish a first connection to a target server through the first proxy;
obtain first content from the target server;
accumulate performance and availability information of the first proxy;
provide the web scraper with a second proxy from the first proxy provider;
establish a second connection to a target server through the second proxy;
obtain second content from the target server;
accumulate performance and availability information of the second proxy;
calculate and configure utilization thresholds of the first proxy provider by evaluating at least one of the performance and availability information of the first proxy or the performance and availability information of the second proxy.
2. The non-transitory computer-readable medium of claim 1 , further comprising periodically reconfiguring the proxy provider's utilization threshold according to: the performance and availability information of the first proxy, the performance and availability information of the second proxy, or both the performance and availability information of the first proxy and the performance and availability information of the second proxy.
3. The non-transitory computer-readable medium of claim 1 , wherein the first proxy is provided to the web scraper by a proxy rotator and wherein the first content is unreadable to the proxy rotator due to network transport layer or application layer encryption.
4. A non-transitory computer readable medium for identifying thresholds of at least a first proxy provider and a second proxy provider, the medium comprising instructions that, when executed by a processor, direct the processor to:
provide a web scraper with at least a first proxy of the first proxy provider;
establish a first connection to a target server through the first proxy of the first proxy provider;
obtain first content from the target server;
accumulate performance and availability information of the first proxy of the first proxy provider;
provide the web scraper with at least a first proxy of the second proxy provider;
establish a second connection to a target server through the second proxy of the second proxy provider;
obtain second content from the target server;
accumulate performance and availability information of the second proxy of the second proxy provider;
calculate and configure utilization the thresholds by evaluating the performance and availability information of the first proxy of the first proxy provider and the performance and availability information of the first proxy of the second proxy provider.
5. The non-transitory computer-readable medium of claim 4 , further comprising periodically reconfiguring the utilization threshold of the first proxy provider or the second proxy provider according to: the performance and availability information of the first proxy of the first proxy provider, the performance and availability information of the first proxy of the second proxy provider, or both the performance and availability information of the first proxy of the first proxy provider and the performance and availability information of the first proxy of the second proxy provider.
6. The non-transitory computer-readable medium of claim 4 , further comprising periodically calculating and configuring a weight of the first proxy provider at any period of time.
7. The non-transitory computer-readable medium of claim 4 , further comprising placing a usage of the first proxy provider on hold, at least for a period of time, when at least one configured maximum threshold is reached.
8. The non-transitory computer-readable medium of claim 4 , wherein the performance and availability information of the first proxy of the first proxy provider or of the first proxy of the second proxy provider collected include at least one of: proxy id, proxy status, proxy response time, or a size of data obtained through the proxy.
9. The non-transitory computer-readable medium of claim 4 , further comprising loading proxy rotator operational settings into a database on disk, a flat file on disk, a database in memory, or other storage media, including proxy provider settings, utilization thresholds, error thresholds, and proxy lists.
10. The non-transitory computer-readable medium of claim 4 , further comprising placing the computing device within a client's network, proxy service provider's network, or any other network.
11. The non-transitory computer-readable medium of claim 4 , wherein the first proxy of the first proxy provider and the first proxy of the second proxy provider are provided to the web scraper by a proxy rotator and wherein the first content and the second content are unreadable to the proxy rotator due to network transport layer or application layer encryption.
12. A computer-implemented method for identifying a threshold of at least one proxy provider, the method comprising:
running a proxy rotator on a computing device connected to a network;
utilizing the proxy rotator to obtain a first proxy from a first proxy provider;
utilizing a web scraper to:
establish a first connection to a target server through the first proxy;
receive first content designated within a first request through the first connection;
accumulate performance and availability information of the first proxy;
utilizing the proxy rotator to obtain a second proxy from the first proxy provider;
utilizing the web scraper to:
establish a second connection to the target server through the second proxy;
receive second content designated within a second request through the second connection;
accumulate performance and availability information of the second proxy;
providing the performance and availability information of the first proxy and the performance and availability information of the second proxy to the proxy rotator;
utilizing the proxy rotator to calculate a weight for the first proxy provider based upon at least one of the performance and availability information of the first proxy or the performance and availability information of the second proxy; and
identifying a threshold of the first proxy provider based on the weight calculated.
13. The method of claim 12 , wherein first proxy provider weight is calculated by the proxy rotator based on the evaluation of the performance and availability information of the first proxy and the performance and availability information of the second proxy of the first proxy provider.
14. The method of claim 12 , further comprising dynamically configuring utilization threshold of the first proxy provider based on the calculated weight.
15. The method of claim 12 , further comprising pre periodically calculating and configuring, via the proxy rotator, the proxy provider weight according to the proxy performance and availability information submitted by the web scraper at any period of time.
16. The method of claim 12 , wherein the proxy rotator stops using a maximized proxy provider, at least for a period of time, when at least one configured maximum threshold is reached.
17. The method of claim 12 , wherein the performance and availability information collected include at least one of: proxy id, proxy status, proxy response time, or a size of data obtained through the proxy.
18. The method of claim 12 , further comprising loading proxy rotator operational settings into a database on disk, a flat file on disk, a database in memory, or other storage media, including proxy provider settings, utilization thresholds, error thresholds, and proxy lists.
19. The method of claim 12 , further comprising placing the proxy rotator within a client's network, proxy service provider's network, or a party providing proxy rotation services.
20. The method of claim 12 , wherein the first content and the second content are unreadable to the proxy rotator due to network transport layer or application layer encryption.