System and method for improving content fetching by selecting tunnel devices
A method for fetching a content from a web server to a client device is disclosed, using tunnel devices serving as intermediate devices. The tunnel device is selected based on an attribute, such as IP Geolocation. A tunnel bank server stores a list of available tunnels that may be used, associated with values of various attribute types. The tunnel devices initiate communication with the tunnel bank server, and stays connected to it, for allowing a communication session initiated by the tunnel bank server. Upon receiving a request from a client to a content and for specific attribute types and values, a tunnel is selected by the tunnel bank server, and is used as a tunnel for retrieving the required content from the web server, using standard protocol such as SOCKS, WebSocket or HTTP Proxy. The client only communicates with a super proxy server that manages the content fetching scheme.
1. A method for operating a group of proxy devices where each of the proxy devices is connected to the Internet, each of the proxy devices is addressable in the Internet using a respective Internet Protocol (IP) address, and each of the proxy devices is selectable to serve as a proxy device for obtaining content from web servers, the method comprising:
storing, at a first server, a list of the IP addresses of the proxy devices in the group;
associating, by the first server, a respective geographical location to each of the proxy devices in the group, using geolocation, wherein multiple proxy devices in the group are associated with a first geographical location;
receiving, by the first server over the Internet from a first device that is configured to serve as a proxy device for obtaining content from web servers, a first message that is indicative of the first device availability to serve as a proxy device;
adding, by the first server, the first device to the group by adding an IP address of the first device to the list, in response to the first message indicating availability of the first device to serve as a proxy device;
associating, by the first server, a respective geographical location to the first device, using geolocation, in response to adding the first device to the group;
receiving, by the first server from a second server over the Internet, the first geographical location;
selecting, by the first server, based on load balancing or performance optimization, a first IP address that is associated with the first geographical location and that is addressing a first proxy device from the list, in response to the receiving of first geographical location;
sending, by the first server to the second server over the Internet, the selected first IP address, in response to the selecting of the first IP address,
using the proxy device addressed by the selected first IP address for obtaining a content from a web server;
receiving, by the first server from the second server over the Internet, a second geographical location;
selecting, by the first server, a second IP address that is associated with the second geographical location and that is addressing a second proxy device from the list, in response to the receiving of second geographical location;
sending, by the first server to the second server over the Internet, the selected second IP address, in response to the selecting of the second IP address; and
using the proxy device addressed by the selected second IP address for obtaining a content from a web server,
wherein each of the geographical locations respectively comprises a respective name or a respective identifier of a respective country, and
wherein the selecting of the first IP address comprises randomly selecting from the multiple IP addresses using one or more random numbers generated by a random number generator.
2. The method according to claim 1 , wherein the list comprises at least 10,000, 20,000, 50,000, 100,000, 200,000, 500,000, 1,000,000, 2,000,000, 5,000,000, or 10,000,000 IP addresses of proxy configured devices.
3. The method according to claim 1 , wherein the selecting from the multiple IP addresses is based on, or uses, a sequential selection, cyclic selection, Last-In-First-Out (LIFO) scheme, First-In-First-Out (FIFO) scheme, or any combination thereof.
4. The method according to claim 1 , wherein each of the geographical locations further respectively comprises a name or an identifier of a respective city, street, Zone Improvement Plan (ZIP) code, timezone, or any combination thereof.
5. The method according to claim 1 , wherein the geolocation is based on W3C Geolocation Application Programming Interface (API).
6. The method according to claim 1 , wherein the geolocation comprises identifying an actual country in which the respective proxy device addressed by the respective IP address is located.
7. The method according to claim 1 , wherein a primary or sole functionality one or more of the proxy devices in the group is to serve as a selected proxy device.
8. A non-transitory computer readable medium containing computer instructions that, when executed by a computer processor, cause the processor to perform the method according to claim 1 .
9. The method according to claim 1 , further comprising storing, operating, or using, by the first server, a server operating system.
10. The method according to claim 1 , wherein each of the proxy devices in the group is associated a single IP address.
11. The method according to claim 1 , further comprising storing, by the first server, at least one respective value associated with at least one attribute type associated with each of proxy devices in the group.
12. The method according to claim 11 , further comprising receiving, by the first server from a second server over the Internet, a first value, and wherein the selecting of the first IP address is based on, or uses, the received first value.
13. The method according to claim 1 , further comprising establishing a connection, by the first server with the first device over the Internet, in response to the adding of the first device to the group.
14. The method according to claim 13 , further comprising establishing a respective connection, by the first server with each of the proxy devices in the group.
15. The method according to claim 13 , wherein a communication between the first server and the first device over the Internet, is over the established connection.
16. The method according to claim 13 , wherein the established connection is a TCP connection using ‘Active OPEN’, ‘Passive OPEN’, or TCP keepalive mechanism.
17. The method according to claim 13 , wherein the established connection uses, or is based on, a Virtual Private Network (VPN).
18. The method according to claim 1 , wherein the random number generator is hardware based.
19. The method according to claim 18 , wherein the random number generator uses thermal noise, shot noise, nuclear decaying radiation, photoelectric effect, quantum phenomena, or any combination thereof.
20. The method according to claim 1 , wherein the random number generator is software based.
21. The method according to claim 20 , wherein the random number generator comprises executing an algorithm for generating pseudo-random numbers.
22. The method according to claim 1 , wherein the first and second servers are owned, operated, or controlled by an entity.
23. The method according to claim 22 , wherein at least one of the proxy devices in the group is owned, operated, or controlled, by the entity.
24. The method according to claim 1 , wherein one or more of the proxy devices in the group is associated with multiple IP addresses.
25. The method according to claim 24 , wherein the one or more of the proxy devices in the group is associated with more than 1,000, 2,000, 5,000, 10,000, 20,000, 50,000 or 100,000 distinct IP addresses.
26. The method according to claim 1 , for use with a first web server that stores a first content that is identified by a first content identifier, the method further comprising:
receiving, by the first proxy device over the Internet, the first content identifier, in response to the selecting of the first IP address;
sending, by the first proxy device to the first web server over the Internet, a first Hypertext Transfer Protocol (HTTP) request that comprises the first content identifier, in response to the receiving of the first content identifier;
receiving, by the first proxy device from the first web server over the Internet, the first content, in response to the sending of the first HTTP request; and
sending, by the first proxy device over the Internet, the first content, in response to the receiving of the first content.
27. The method according to claim 26 , wherein the first web server further stores a second content that is identified by a second content identifier, the method further comprising:
receiving, by the second proxy device over the Internet, the second content identifier, in response to the selecting of the second IP address;
sending, by the second proxy device to the first web server over the Internet, a second Hypertext Transfer Protocol (HTTP) request that comprises the second content identifier, in response to the receiving of the second content identifier;
receiving, by the second proxy device from the first web server over the Internet, the second content, in response to the sending of the second HTTP request; and
sending, by the second proxy device over the Internet, the second content, in response to the receiving of the second content.
28. The method according to claim 26 , for use with a second web server that stores a second content that is identified by a second content identifier, the method further comprising:
receiving, by the second proxy device over the Internet, the second content identifier, in response to the selecting of the second IP address;
sending, by the second proxy device to the second web server over the Internet, a second Hypertext Transfer Protocol (HTTP) request that comprises the second content identifier, in response to the receiving of the second content identifier;
receiving, by the second proxy device from the second web server over the Internet, the second content, in response to the sending of the second HTTP request; and
sending, by the second proxy device over the Internet, the second content, in response to the receiving of the second content.
29. The method according to claim 26 , wherein the first content comprises a web-page or a web-site, and wherein the first content identifier comprises a Uniform Resource Identifier (URI) or Uniform Resource Locator (URL).
30. The method according to claim 26 , wherein the first HTTP request comprises a HTTP Secure (HTTPS) request.
31. The method according to claim 1 , further comprising removing a third proxy device from the group by removing an IP address of the third proxy device from the list.
32. The method according to claim 31 , further comprising receiving, by the first server over the Internet from the third proxy device, a third message that is indicative of the third proxy device availability to serve as a proxy device.
33. The method according to claim 32 , wherein the removing is in response to the third message indicating non-availability of the third proxy device to serve as a proxy device.