System and method for improving content fetching by selecting tunnel devices
A method for fetching a content from a web server to a client device is disclosed, using tunnel devices serving as intermediate devices. The tunnel device is selected based on an attribute, such as IP Geolocation. A tunnel bank server stores a list of available tunnels that may be used, associated with values of various attribute types. The tunnel devices initiate communication with the tunnel bank server, and stays connected to it, for allowing a communication session initiated by the tunnel bank server. Upon receiving a request from a client to a content and for specific attribute types and values, a tunnel is selected by the tunnel bank server, and is used as a tunnel for retrieving the required content from the web server, using standard protocol such as SOCKS, WebSocket or HTTP Proxy. The client only communicates with a super proxy server that manages the content fetching scheme.
1. A method for managing proxy servers, for use with a group of proxy servers, and for use with a database that associate to each of the proxy servers in the group a service level value that is responsive to a former measurement of the respective proxy server activity, the method comprising:
providing, to a client device, a first proxy server, wherein the first proxy server stores a first list of IP addresses;
establishing a first connection to a first target server through the first proxy server;
obtaining a first content from the first target server using a first IP address selected from the first list;
accumulating performance and availability information of the first proxy server;
providing, to the client device, a second proxy server, wherein the second proxy server stores a second list of IP addresses;
establishing a second connection to a second target server through the second proxy server;
obtaining a second content from the second target server using a second IP address selected from the second list;
accumulating performance and availability information of the second proxy server; and
calculating and configuring utilization thresholds by evaluating at least one of the performance and availability information of the first proxy server or the performance and availability information of the second proxy server,
wherein the providing of the first proxy server or the providing of the second proxy server to the client device is based on, uses, or responsive to, the respective service level values, and
wherein the accumulating performance and availability information of the first or second proxy servers, comprises measuring or estimating of Bit Error Rate (BER), Received Signal Strength Indicator (RSSI), Round-Trip delay Time (RTT), Bandwidth (BW), Packet Loop Ratio (PLR), or Cyclic Redundancy Check (CRC), that is associated with the respective proxy server or with the respective obtaining of the respective content.
2. The method according to claim 1 , wherein the first target server comprises, or consists of, a first web server for serving the content accessible through HTTP/HTTPS protocols, and wherein the second target server comprises, or consists of, a second web server for serving the content accessible through HTTP/HTTPS protocols.
3. The method according to claim 1 , wherein the same server serves as the first and second target server.
4. The method according to claim 1 , wherein the first proxy server and the second proxy server are operated by a first proxy provider.
5. The method according to claim 1 , wherein the first proxy server is operated by a first proxy provider and wherein the second proxy server is operated by a second proxy provider that is distinct from the first proxy provider.
6. The method according to claim 1 , further comprising selecting the first IP address from the first list, and selecting the second IP address from the second list.
7. The method according to claim 1 , wherein the providing of the first proxy server comprises selecting of the first proxy server from the group based on the respective service level value, and wherein the providing of the second proxy server comprises selecting of the second proxy server from the group based on the respective service level value.
8. The method according to claim 1 , wherein the accumulating performance and availability information of the first or second proxy servers, or the evaluating, comprises updating the service level values that are respectively associated with the first and second proxy servers in the database.
9. The method according to claim 1 , wherein the measuring further comprises measuring or estimating of an amount of data fetched in the respective obtaining, measuring or estimating of the number of IP addresses in the respective list, estimating a physical location of the respective proxy server, measuring or estimating of available bandwidth or throughput of the respective proxy server or of the respective obtaining, measuring or estimating of communication errors or packets loss associated with the respective proxy server or with the respective obtaining, measuring or estimating of number of hops to the respective target server from the respective proxy server, or measuring or estimating of transfer delay time associated with the respective proxy server or with the respective obtaining.
10. The method according to claim 1 , further for periodically updating the service level values associated with at least one of the first and second proxy servers, based on periodically estimating or measuring of the performance and availability information of respectively the at least one of the first and second proxy servers.
11. The method according to claim 1 , further comprising removing a proxy server from the database based on a respective service level value or based on a measurement of the respective proxy server activity, availability, or performance.
12. The method according to claim 1 , further for periodically estimating or measuring of the performance and availability information of at least one of the first and second proxy servers.
13. The method according to claim 1 , further comprising periodically reconfiguring the proxy provider's utilization threshold according to: the performance and availability information of the second proxy, or both the performance and availability information of the first proxy and the performance and availability information of the second proxy.
14. A non-transitory computer readable medium that comprises instructions that, when executed by a processor, direct the processor to execute the steps of claim 1 .
15. The method according to claim 1 , wherein the performance and availability information of the first or second proxy server comprises a respective proxy server identification, a respective proxy server status, a respective proxy server, or a size of data obtained through the respective proxy.
16. The method according to claim 1 , further for providing anonymity, wherein each of the first and second proxy servers consists of, comprises, uses, or is integrated with, a proxy server that is a HTTP proxy server, a forward proxy server, a reverse proxy server, transparent proxy server, or a non-transparent proxy server.
17. The method according to claim 1 , further for identifying a threshold of at least one proxy provider, wherein each of the first and second proxy servers is configured to acting as an intermediary for requests from clients seeking resources from other servers.
18. The method according to claim 17 , wherein each of the first and second proxy servers is configured to receive a request for service from a client device, to evaluate the request for service, and to forwards the request through the actual target resource or resources that contain the actual content.
19. The method according to claim 18 , wherein each of the first and second proxy servers is further configured to forward a received content to the requesting client device.
20. The method according to claim 18 , wherein the service comprises a file, a connection, a web page, or a resource available from a server.
21. The method according to claim 18 , wherein each of the first and second proxy servers is configured to have full visibility into the respective received content.
22. The method according to claim 18 , wherein each of the first and second proxy servers is configured to serve as an intermediary device in encrypted or HTTPS sessions, so that it is not aware of the received content.
23. The method according to claim 1 , further comprising storing and executing, by the client device, an automated computer program or script that gathers data or content from a network for the purpose of replicating content or for data analysis.
24. The method according to claim 23 , wherein the client device is further configured to communicate using Hypertext Transfer Protocol (HTTP), HTTP Secure (HTTPS), or File Transfer Protocol (FTP) for accessing web resources or for employing a proxy server as an intermediary.
25. The method according to claim 23 , wherein the client device is further configured to store and execute a web browser application for downloading a web page or objects stored in a web browser.
26. The method according to claim 1 , further for use with a device that stores the IP addresses of the first and second proxy server, wherein the providing of the client device with the first proxy server comprises sending from the device to the client device the IP address of the first proxy server, and wherein the providing of the client device with the second proxy server comprises sending from the device to the client device the IP address of the second proxy server.
27. The method according to claim 26 , wherein the device is unaware of the first and second content.