Smart proxy rotator
The task, logic of HTTP/HTTPS session statistics interception and collection is moved to the client side instead of the proxy layer. Encrypted HTTPS tunnel is terminated at the client end, making the actual content or data in transit invisible to both proxies and the smart proxy rotator (SPR). Client's scraping software has a plug-in installed that expands its functionality. HTTP/HTTPS session quality metrics are intercepted and collected at the client side, then sent to the SPR. Proxy usage mark “can be used” is obtained from the SPR for the currently analyzed proxy, based on the results of metrics analysis.
1. A computer-implemented method for identifying a threshold of a proxy provider, the method comprising:
running a proxy rotator on a computing device connected to a network;
obtaining a proxy from a proxy provider by the proxy rotator;
utilizing a web scraper to:
establish a connection to a target server through the proxy;
receive content designated within a request through the connection;
provide performance and availability information of the proxy to the proxy rotator;
utilizing the proxy rotator to calculate a weight for the proxy provider; and
identifying a threshold of the proxy provider based on the calculated weight;
wherein the proxy rotator stops using a maximized proxy provider, at least for a period of time, when at least one configured maximum threshold is reached;
wherein the at least one configured maximum threshold is reached when a maximum number of concurrent connections is reached, a maximum error threshold is reached, or both the maximum number of concurrent connections is reached and the maximum error threshold is reached;
wherein the content is unreadable to the proxy rotator due to network transport layer or application layer encryption.
2. The method of claim 1 , wherein the web scraper accumulates the performance and availability information from a plurality of proxies before providing the performance and availability information to the proxy rotator.
3. The method of claim 1 , wherein the proxy provider weight is calculated by the proxy rotator based on the evaluation of the performance and availability information from the proxy.
4. The method of claim 1 further comprising dynamically configuring utilization threshold of the proxy provider based on the calculated weight.
5. The method of claim 1 , further comprising periodically calculating and configuring, via the proxy rotator, the proxy provider weight according to the performance and availability information of the proxy provided by the web scraper at any period of time.
6. The method of claim 1 , wherein the performance and availability information collected include at least one of: proxy id, proxy status, proxy response time, or a size of data obtained through the proxy.
7. The method of claim 1 , further comprising loading proxy rotator operational settings into a database on disk, a flat file on disk, a database in memory, or other storage media, including proxy provider settings, utilization thresholds, error thresholds, and proxy lists.
8. The method of claim 1 , further comprising placing the proxy rotator within a client's network, proxy service provider's network, or a party providing proxy rotation services.
9. A non-transitory computer-readable medium for identifying a threshold of a proxy provider, the medium comprising instructions that, when executed by a processor, direct the processor to:
run a proxy rotator on a computing device connected to a network;
obtain a proxy from a proxy provider by the proxy rotator;
utilize a web scraper to:
establish a connection to a target server through the proxy;
receive content designated within a request through the connection;
provide performance and availability information of the proxy to the proxy rotator;
utilize the proxy rotator to calculate a weight for the proxy provider;
identify a threshold of the proxy provider based on the calculated weight; and
placing a usage of the proxy provider on hold, at least for a period of time, when at least one configured maximum threshold is reached;
wherein the at least one configured maximum threshold is reached when a maximum number of concurrent connections is reached, a maximum error threshold is reached, or both the maximum number of concurrent connections is reached and the maximum error threshold is reached.
10. The non-transitory computer-readable medium of claim 9 , further comprising periodically calculating and configuring a weight of the proxy provider at any period of time.
11. The non-transitory computer-readable medium of claim 9 , wherein the performance and availability information of the proxy collected include at least one of: proxy id, proxy status, proxy response time, or a size of data obtained through the proxy.
12. The non-transitory computer-readable medium of claim 9 , further comprising loading proxy rotator operational settings into a database on disk, a flat file on disk, a database in memory, or other storage media, including proxy provider settings, utilization thresholds, error thresholds, and proxy lists.
13. The non-transitory computer-readable medium of claim 9 , further comprising placing the computing device within a client's network, proxy service provider's network, or any other network.
14. The non-transitory computer-readable medium of claim 9 , wherein the content is unreadable to the proxy rotator due to network transport layer or application layer encryption.
15. A computer-implemented method for identifying a threshold of a proxy provider, the method comprising:
running a proxy rotator on a computing device connected to a network;
obtaining a proxy from a proxy provider by the proxy rotator;
utilizing a web scraper to:
establish a connection to a target server through the proxy;
receive content designated within a request through the connection;
provide performance and availability information of the proxy to the proxy rotator;
utilizing the proxy rotator to calculate a weight for the proxy provider; and
identifying a threshold of the proxy provider based on the calculated weight;
wherein the content is unreadable to the proxy rotator due to network transport layer or application layer encryption.