System and method for improving content fetching by selecting tunnel devices
A method for fetching a content from a web server to a client device is disclosed, using tunnel devices serving as intermediate devices. The tunnel device is selected based on an attribute, such as IP Geolocation. A tunnel bank server stores a list of available tunnels that may be used, associated with values of various attribute types. The tunnel devices initiate communication with the tunnel bank server, and stays connected to it, for allowing a communication session initiated by the tunnel bank server. Upon receiving a request from a client to a content and for specific attribute types and values, a tunnel is selected by the tunnel bank server, and is used as a tunnel for retrieving the required content from the web server, using standard protocol such as SOCKS, WebSocket or HTTP Proxy. The client only communicates with a super proxy server that manages the content fetching scheme.
1. A method for operating a database that associates a respective value indicative of performance and availability of a group of servers, each of the servers in the group stores a respective distinct list of Internet Protocol (IP) addresses, the method comprising:
identifying a first content that comprises a first web page or a part thereof;
selecting, from the group of servers, based on the associated value in the database and on load balancing, a first server that is operated by a first proxy provider and that stores a first list of IP addresses;
selecting, a first IP address from the first list, by using a random selection, a sequential selection, or a cycling selection;
sending, to a first client device that is addressed by the selected first IP address over the Internet, a first request for the first content;
receiving, from the first client device over the Internet, in response to the sending of the first request, the first content;
measuring, a first delay associated with the receiving of the first content;
updating, in the database, a value associated with the first server, in response to, or based on, the measured first delay;
identifying a second content that comprises a second web page or a part thereof;
selecting, from the group of servers, based on the associated value in the database and on load balancing, a second server that stores a second list of IP addresses, and that is operated by a second proxy provider that is operated by a second proxy provider that is distinct from the first proxy provider;
selecting, a second IP address from the second list, by using a random selection, a sequential selection, or a cycling selection;
sending, to a second client device that is addressed by the selected second IP address over the Internet, a second request for the second content;
receiving, from the second client device over the Internet, in response to the sending of the second request, the second content;
measuring, a second delay associated with the receiving of the second content; and
updating, in the database, a value associated with the second server, in response to, or based on, the measured second delay,
periodically reconfiguring proxy provider's utilization threshold according to the performance and availability information of a server from the group wherein the performance and availability information of the first or second server comprises a respective server identification, a respective server status, a respective server activity, a size of data obtained through the respective server, or any combination thereof,
wherein the measuring of the first or second delay further comprises measuring or estimating of Round-Trip delay Time (RTT), Bandwidth (BW), or Packet Loop Ratio (PLR).
2. The method according to claim 1 , wherein the first or second request comprises a Hypertext Transfer Protocol (HTTP) or HTTP Secure (HTTPS) request.
3. The method according to claim 1 , wherein the first or second content comprises a part of, or whole of, audio data, voice data, multimedia data, video data, a computer program, or any combination thereof.
4. The method according to claim 1 , further comprising identifying of the first content by a requesting first client device, and identifying of the second content by the requesting second client device.
5. The method according to claim 1 , further comprising:
selecting, from the group, based on the associated value in the database, a third server that is operated by a third proxy provider and that stores a third list of IP addresses;
selecting, from the third list, by using a random selection, a sequential selection, or a cycling selection;
sending, to a third client device that is addressed by the selected first IP address over the Internet, a third request for a third content;
receiving, from the third client device over the Internet, in response to the sending of the third request, the third content;
measuring, a third delay associated with the receiving of the third content; and
updating, in the database, a value associated with the third server, in response to, or based on, the measured third delay.
6. The method according to claim 1 , wherein each of the values in the database is responsive to a respective measurement of the respective server activity.
7. The method according to claim 1 , wherein the measuring of the first or second delay further comprises measuring or estimating of Bit Error Rate (BER), Received Signal Strength Indicator (RSSI), Cyclic Redundancy Check (CRC), or any combination thereof.
8. A non-transitory computer readable medium that comprises instructions that, when executed by a processor, direct the processor to execute the steps of claim 1 .
9. The method according to claim 1 , further for providing anonymity, wherein each of the first and second servers consists of, comprises, uses, or is integrated with, a proxy server that is a HTTP proxy server, a forward proxy server, a reverse proxy server, transparent proxy server, or a non-transparent proxy server.
10. The method according to claim 1 , wherein the first or second client device is configured to serve as an intermediary device in encrypted sessions.
11. The method according to claim 1 , further comprising periodically updating the values associated with the first and second servers, based on periodically estimating or measuring of the performance and availability information of respectively the at least one of the first and second servers.
12. The method according to claim 1 , wherein each of IP addresses in the first or second list is associated with a respective client device that is configurable to serve as a respective proxy device for obtaining a respective content from a respective web server.
13. The method according to claim 1 , further comprising removing a server from the group by removing from the database, based on a respective value or based on a measurement of the respective server activity, availability, or performance.
14. The method according to claim 3 , wherein the first content is identified by a first Uniform Resource Locator (URL), and wherein the second content is identified by a second URL.
15. The method according to claim 14 , wherein the first request comprises the first URL, and wherein the first request comprises the second URL.
16. The method according to claim 1 , wherein the first content is stored in a first web server, and wherein the second content is stored in a second web server.
17. The method according to claim 16 , further comprising:
sending, by the first client device to the first web server over the Internet, a third request for the first content, in response to the sending of the first request to the first client device;
receiving, by the first client device from the first web server over the Internet, the first content, in response to the sending of the third request;
sending, by the second client device to the second web server over the Internet, a fourth request for the second content, in response to the sending of the second request to the second client device; and
receiving, by the second client device from the second web server over the Internet, the second content, in response to the sending of the fourth request,
wherein the receiving from the first client device over the Internet of the first content, is in response to the receiving of the first content by the first client device, and
wherein the receiving from the second client device over the Internet of the second content, is in response to the receiving of the second content by the second client device.
18. The method according to claim 1 , wherein the first content is stored in a web server, and wherein the second content is stored in the web server.
19. The method according to claim 18 , further comprising:
sending, by the first client device to the web server over the Internet, a third request for the first content, in response to the sending of the first request to the first client device;
receiving, by the first client device from the web server over the Internet, the first content, in response to the sending of the third request;
sending, by the second client device to the web server over the Internet, a fourth request for the second content, in response to the sending of the second request to the second client device; and
receiving, by the second client device from the web server over the Internet, the second content, in response to the sending of the fourth request,
wherein the receiving from the first client device over the Internet of the first content, is in response to the receiving of the first content by the first client device, and
wherein the receiving from the second client device over the Internet of the second content, is in response to the receiving of the second content by the second client device.
20. The method according to claim 1 , wherein the identifying of the first content comprises receiving from a requesting client device a first identifier of the first content, and wherein the identifying of the second content comprises receiving from a requesting client device a second identifier of the second content.
21. The method according to claim 20 , further comprising sending, to the requesting client device over the Internet, the first content in response to the receiving of the first content from the second client device over the Internet, and sending, to the requesting client device over the Internet, the second content in response to the receiving of the second content from the second client device over the Internet.
22. The method according to claim 1 , wherein each of the first and second servers is configured to receive a request for service from a requesting client device, to evaluate the request for service, and to forwards the request through an actual web resource that contains the actual requested content.
23. The method according to claim 22 , wherein each of the first and second servers is further configured to forward a received content to the requesting client device.
24. The method according to claim 22 , wherein the service comprises a file, a connection, a web page, or a resource available from a server.
25. The method according to claim 22 , wherein each of the first and second servers is configured to have full visibility into the respective received content.
26. The method according to claim 1 , further comprising storing and executing, by a requesting client device, an automated computer program or script that gathers data or content from a network for replicating content or for data analysis.
27. The method according to claim 26 , wherein the requesting client device is further configured to communicate using Hypertext Transfer Protocol (HTTP), HTTP Secure (HTTPS), or File Transfer Protocol (FTP) for accessing web resources or for employing a proxy server as an intermediary.
28. The method according to claim 26 , wherein the requesting client device is further configured to store and execute a web browser application for downloading a web page or objects stored in a web browser.
29. The method according to claim 26 , wherein the selecting of the first or second server is performed by the requesting client device.
30. The method according to claim 26 , wherein the selecting of the first or second IP address is performed by the requesting client device.
31. The method according to claim 1 , wherein the measuring of the first or second delay further comprises detecting or measuring error.
32. The method according to claim 31 , wherein the detecting or measuring error comprises measuring or estimating of an amount of data fetched in the respective receiving, measuring or estimating of a number of IP addresses in the respective list, estimating a physical location of the respective server, measuring or estimating of available bandwidth or throughput of the respective server or of the respective receiving, measuring or estimating of communication errors or packets loss associated with the respective server or with the respective obtaining, measuring or estimating of number of hops to a respective web server from the respective server, measuring or estimating of transfer delay time associated with the respective server or with the respective receiving, or any combination thereof.