Managing and selecting proxy devices by multiple servers
A method for fetching a content from a web server to a client device is disclosed, using tunnel devices serving as intermediate devices. The tunnel device is selected based on an attribute, such as IP Geolocation. A tunnel bank server stores a list of available tunnels that may be used, associated with values of various attribute types. The tunnel devices initiate communication with the tunnel bank server, and stays connected to it, for allowing a communication session initiated by the tunnel bank server. Upon receiving a request from a client to a content and for specific attribute types and values, a tunnel is selected by the tunnel bank server, and is used as a tunnel for retrieving the required content from the web server, using standard protocol such as SOCKS, WebSocket or HTTP Proxy. The client only communicates with a super proxy server that manages the content fetching scheme.
1. A method for managing multiple client devices that are configured to serve as proxy devices for anonymously obtaining content and that are addressed by respective multiple Internet Protocol (IP) addresses, by updating and operating a group of servers, each of the servers stores a respective database that comprises IP addresses of at least part of the multiple client devices, the method comprising:
establishing, by a first server selected from the group of servers with a first client device over the Internet, a first secured connection that uses a first secured connection, in response to a communication initiated by the first client device;
receiving, by the first server from the first client device over the first secured connection, a first status message indicative of an availability of a first resource that is used by the first client device upon serving as a proxy device for obtaining content from a web server;
adding, by the first server, a first IP address of the first client device to the respective database in response to the receiving of the first status message;
establishing, by the first server with a second client device over the Internet, a second secured connection that uses a second secured connection, in response to a communication initiated by the second client device;
receiving, by the first server from the second client device over the second secured connection, a second status message indicative of an availability of a second resource that is used by the second client device upon serving as a proxy device for obtaining content from a web server;
adding, by the first server, a second IP address of the second client device to the respective database in response to the receiving of the second status message;
associating, a respective country to each of the IP addresses in the databases in the servers in the group of servers, using geolocation of the respective IP addresses, for identifying the actual country in which the respective proxy device is located;
storing, in the databases in the servers in the group of servers, the associated respective countries associated with the respective IP addresses;
receiving, by the first server from a first requesting device over the Internet, a first identifier of a first country;
selecting, by the first server from the respective database in the first server, a third IP address that is associated with the first country, in response to the receiving of the first identifier;
sending, by the first server to a first requesting device over the Internet, the selected third IP address in response to the selecting of the third IP address;
receiving, by a second server selected from the group of servers, from a second requesting device over the Internet, a second identifier of a second country;
selecting, by the second server from the respective database in the first server, a fourth IP address that is associated with the second country, in response to the receiving of the second identifier; and
sending, by the second server to a second requesting device over the Internet, the selected fourth IP address in response to the selecting of the fourth IP address wherein the selecting of the first or second server from the group of servers utilize a load balancing scheme or performance optimization.
2. The method according to claim 1 , wherein the third and fourth IP addresses are associated with different countries.
3. The method according to claim 1 , wherein the third and fourth IP addresses are associated with the same country.
4. The method according to claim 1 , further comprising:
establishing, by the first server with a third client device over the Internet, a third secured connection that uses a third secured connection, in response to communication initiated by the third client device;
receiving, by the first server from the third client device over the third secured connection, a third status message indicative of an availability of the first resource that is used by the third client device upon serving as a proxy device for obtaining content from a web server; and
adding, by the first server, a third IP address of the third client device to the respective database in response to the receiving of the third status message.
5. The method according to claim 1 , further comprising:
receiving, by the first server from a third requesting device over the Internet, a third identifier of a third country;
selecting, by the first server from the respective database in the first server, a fifth IP address that is associated with the third country, in response to the receiving of the third identifier; and
sending, by the first server to a third requesting device over the Internet, the selected fifth IP address in response to the selecting of the fifth IP address.
6. The method according to claim 1 , wherein the proxy server that is a HTTP proxy server, a forward proxy server, a reverse proxy server, transparent proxy server, a non-transparent proxy server, or any combination thereof.
7. A non-transitory computer readable medium containing computer instructions that, when executed by a computer processor, cause the processor to perform all of the steps of claim 1 .
8. The method according to claim 1 , further comprising selecting the first server from the group.
9. The method according to claim 8 , wherein the receiving, of the first identifier is in response to selecting of the first server from the group.
10. The method according to claim 8 , wherein the selecting is by the first requesting device.
11. The method according to claim 8 , further comprising selecting the second server from the group.
12. The method according to claim 11 , wherein the receiving, of the second identifier is in response to the selecting of the second server from the group.
13. The method according to claim 11 , wherein the selecting is by the second requesting device.
14. The method according to claim 1 , further comprising selecting the first or second server from the group.
15. The method according to claim 14 , wherein the geolocation comprises IP geolocation that is based on W3C Geolocation API.
16. The method according to claim 14 , wherein the geolocation comprises identifying the actual country in which the respective device addressed by the respective IP address is located.
17. The method according to claim 14 , wherein the associating further comprises associating with a region, a city, a street, or a ZIP code, in the respective associated country.
18. The method according to claim 14 , wherein the selecting of the first or second server from the group comprises randomly selecting from the group.
19. The method according to claim 18 , wherein the randomly selecting is based on using one or more random numbers generated by a random number generator.
20. The method according to claim 19 , wherein the random number generator comprises hardware based and uses thermal noise, shot noise, nuclear decaying radiation, photoelectric effect, or quantum phenomena.
21. The method according to claim 19 , wherein the random number generator is software based and is based on executing an algorithm for generating pseudo-random numbers.
22. The method according to claim 14 , wherein each one of the servers in the group is associated with a respective value relating to an attribute type.
23. The method according to claim 22 , wherein the selecting of the first or second server from the group comprises selecting based on, or according to, the respective value of the selected server.
24. The method according to claim 14 , wherein the selecting of the first or second server from the group is based on, or uses, a load balancing or performance optimization.
25. The method according to claim 24 , wherein the selecting of the first or second server from the group is based on, or uses, a sequential selection, cyclic selection, Last-In-First-Out (LIFO) scheme, First-In-First-Out (FIFO) scheme, or any combination thereof.
26. The method according to claim 1 , further for use with a virtualization, wherein at least one of the steps is executed as part of a virtualized application as part of a Virtual Machine (VM), or wherein the first or second server is at least in part implemented as virtual hardware.
27. The method according to claim 26 , for use with a host computer that implement the VM, wherein the method further comprising executing, by the host computer, a hypervisor or a Virtual Machine Monitor (VMM), and wherein the virtualized application or hardware uses or interfaces virtual hardware.
28. The method according to claim 26 , wherein the virtualization includes, is based on, or uses, full virtualization, para-virtualization, or hardware assisted virtualization.
29. The method according to claim 1 , wherein the first or second secured connection is based on, uses, or is compatible with, Socket Secure (SOCKS) protocol or connection, wherein the first or second server serves as a SOCKS server.
30. The method according to claim 29 , wherein the SOCKS protocol or connection is according to, based on, or is compatible with, SOCKS4, SOCKS4a, or SOCKS5.
31. The method according to claim 29 , wherein the SOCKS protocol or connection is according to, based on, or is compatible with, IETF RFC 1928, IETF RFC 1929, IETF RFC 1961, or IETF RFC 3089.
32. The method according to claim 1 , wherein at least one of the databases comprises a list of at least 2, 3, 4, 5, 8, 10, 20, 30, 50, 80, 100, 120, 150, 200, 300, 500, 1,000, 2,000, 3,000, 5,000, 10,000, 20,000, 30,000, 50,000, or 100,000 distinct IP addresses.
33. The method according to claim 32 , wherein at least one of the databases comprises a list of less than 3, 4, 5, 8, 10, 20, 30, 50, 80, 100, 120, 150, 200, 300, 500, 1,000, 2,000, 3,000, 5,000, 10,000, 20,000, 30,000, 50,000, 100,000 or 200,000 distinct IP addresses.