Software development kit (SDK) for selecting and implementing client devices as proxies
A method for fetching a content from a web server to a client device is disclosed, using tunnel devices serving as intermediate devices. The tunnel device is selected based on an attribute, such as IP Geolocation. A tunnel bank server stores a list of available tunnels that may be used, associated with values of various attribute types. The tunnel devices initiate communication with the tunnel bank server, and stays connected to it, for allowing a communication session initiated by the tunnel bank server. Upon receiving a request from a client to a content and for specific attribute types and values, a tunnel is selected by the tunnel bank server, and is used as a tunnel for retrieving the required content from the web server, using standard protocol such as SOCKS, WebSocket or HTTP Proxy. The client only communicates with a super proxy server that manages the content fetching scheme.
1. A computed implemented method comprising:
storing, by a first server, a database that comprises a list of Internet Protocol (IP) addresses of a plurality of devices, each one of the plurality of devices includes a respective copy of a Software Development Kit (SDK) that is configured to perform a task that comprises a proxy function;
receiving, by the first server over the Internet, from a first device that is addressed using a first Internet Protocol (IP) address and that includes a respective copy of the Software Development Kit (SDK) configured to perform the task, a first message;
establishing a Transmission Control Protocol (TCP) connection over the Internet between the first server and the first device in response to the receiving of the first message;
updating, by the first server, the stored list in the database, by adding the first IP address to the stored list, in response to the receiving of the first message;
receiving, by the first server over the Internet, from a second device that is addressed using second Internet Protocol (IP) address and that includes a respective copy of the Software Development Kit (SDK) configured to perform the task, a second message;
establishing a Transmission Control Protocol (TCP) connection over the Internet between the first server and the second device in response to the receiving of the second message;
updating, by the first server, the stored list in the database, by adding the second IP address to the stored list, in response to the receiving of the second message;
receiving, by the first server over the Internet, from third device that is addressed using a third Internet Protocol (IP) address and that includes a respective copy of the Software Development Kit (SDK) configured to perform the task, a third message;
establishing a Transmission Control Protocol (TCP) connection over the Internet between the first server and the third device in response to the receiving of the third message;
updating, by the first server, the stored list in the database, by adding the third IP address to the stored list, in response to the receiving of the third message;
randomly selecting, by the first server, the first IP address from the list, based on a first value associated with the first IP address;
sending, by the first server to the first device over the Internet, a first request to perform the task that comprises using the first device as a proxy device, in response to the selecting of the first IP address;
randomly selecting, by the first server, the second IP address from the list, based on a second value associated with the second IP address;
sending, by the first server to the second device over the Internet, a second request to perform the task that comprises using the first device as a proxy device, in response to the selecting of the second IP address;
randomly selecting, by the first server, the third IP address from the list, based on a third value associated with the third IP address; and
sending, by the first server to the third device over the Internet, in response to the selecting of the third IP address, third request to perform the task that comprises using the first device as a proxy device.
2. The method according to claim 1 , wherein the SDK comprises computer instructions that, when executed by a processor in a device, cause the device to serve as an intermediate device for obtaining a content from a web server.
3. The method according to claim 1 , further preceded by initiating, by the first device, a sending of the first message to the first server over the Internet; initiating, by the second device, a sending of the second message to the first server over the Internet; and initiating, by the third device, a sending of the third message to the first server over the Internet.
4. The method according to claim 1 , wherein the established connection each one of the plurality of devices uses ‘Active OPEN’ and ‘Passive OPEN’, wherein the established connection each one of the plurality of devices comprises or uses a TCP keepalive mechanism, or wherein the established connection each one of the plurality of devices comprises or uses a Virtual Private Network (VPN).
5. The method according to claim 1 , wherein any communication over the Internet between the first server and advice of the plurality of devices is based on, uses, or is compatible with, a Transmission Control Protocol over Internet Protocol (TCP/IP) protocol or connection, or wherein the communication over the Internet between the first server and advice of the plurality of devices is based on, uses, or is compatible with, a Hypertext Transfer Protocol (HTTP) or Hypertext Transfer Protocol Secure (HTTPS) protocol or connection.
6. The method according to claim 1 , wherein the communication over the Internet between the first server and a device of the plurality of devices is based on, uses, or is compatible with, a HTTP Proxy protocol or connection.
7. The method according to claim 1 , wherein the selecting of the first, second, or third IP address is based on load balancing.
8. The method according to claim 1 , wherein the selecting of the first, second, or third IP address is further based on, or further uses deterministic selection.
9. The method according to claim 1 , wherein the selecting of the first, second, or third IP address is based on, or uses, Last-In-First-Out(LIFO) or First-In-First-Out (FIFO) scheme.
10. The method according to claim 1 , wherein the selecting of the first, second, or third IP address is based on, or using, sequential or cyclic selection.
11. The method according to claim 1 , further comprising, storing, operating, or using, by the first server, a server operating system.
12. The method according to claim 1 , further comprising: receiving, by the first server over the Internet, from a fourth device that is addressed using fourth Internet Protocol (IP) address and that includes a respective copy of the Software Development Kit (SDK) configured to perform the task, a fourth message; and
updating, by the first server, the stored list, by adding the fourth IP address to the stored list, in response to the receiving of the fourth message.
13. The method according to claim 12 , further comprising:
selecting, by the first server, the fourth IP address from the list; and
sending, by the first server to the fourth device over the Internet, in response to the selecting of the fourth IP address, fourth request to perform the task.
14. The method according to claim 1 , wherein the sending of the first request is over the established connection.
15. The method according to claim 14 , wherein the established connection uses ‘Active OPEN’ and ‘Passive OPEN’.
16. The method according to claim 14 , wherein the established connection comprises or uses a TCP keepalive mechanism.
17. The method according to claim 14 , wherein the established connection comprises or uses a Virtual Private Network (VPN).
18. The method according to claim 1 , wherein the communication over the Internet between the first server and a device of the plurality of devices is based on, uses, or is compatible with, a Socket Secure (SOCKS) protocol or connection.
19. The method according to claim 18 , wherein the SOCKS protocol or connection is according to, based on, or is compatible with, SOCKS4, SOCKS4a, or SOCKS5.
20. The method according to claim 18 , wherein the SOCKS protocol or connection is according to, based on, or is compatible with, IETF RFC 1928, IETF RFC 1929, IETF RFC 1961, or IETF RFC 3089.
21. The method according to claim 1 , over the Internet between the first server and a device of the plurality of devices is based on, uses, or is compatible with, WebSocket (ws) or WebSocket Secure (wss) protocol or connection.
22. The method according to claim 21 , wherein the respective WebSocket (ws) or WebSocket Secure (wss) protocol or connection is according to, based on, or is compatible with, IETF RFC 6455.
23. The method according to claim 1 , wherein the list comprises at least 1, 2, 5, 8, 10, 12, 15, 20, 20, 30, 50, 80, 100, 120, 150, 200, 500, 1,000, 2,000, 5,000, or 10,000 IP addresses.
24. The method according to claim 23 , wherein the list comprises less than 2, 3, 4, 5, 8, 10, 12, 15, 20, 20, 30, 50, 80, 100, 120, 150, 200, 500, 1,000, 2,000, 5,000, 10,000 or 20,000 IP addresses.
25. The method according to claim 1 , further comprising storing, by the first server, a respective geographical location associated with each of the IP addresses in the list.
26. The method according to claim 25 , wherein each of the geographical location comprises a respective country or city.
27. The method according to claim 25 , further comprising associating the respective geographical location with each of the IP addresses in the list.
28. The method according to claim 27 , wherein the associating is based on IP geolocation.
29. The method according to claim 28 , wherein the geolocation is based on W3C Geolocation API, or for use with database stored in the first server that associates IP addresses to geographical locations.
30. The method according to claim 1 , wherein the first message comprises the first value, wherein the second message comprises the second value.
31. The method according to claim 30 , wherein the first value comprises a status or state of the first device, wherein the second value comprises a status or state of the second device, and wherein the third value comprises a status or state of the third device.
32. The method according to claim 30 , wherein each of the values respectively comprises a name or an identifier of an Internet Service Provider (ISP) or an Autonomous System Number (ASN) number.
33. The method according to claim 30 , wherein each of the values respectively corresponds to hardware of software of the respective device.
34. The method according to claim 30 , wherein each of the values respectively corresponds to the respective device being stationary or portable.
35. The method according to claim 30 , wherein each of the values respectively corresponds to the respective device connection to the Internet.
36. The method according to claim 30 , wherein each of the values respectively corresponds to a bandwidth (BW) or Round-Trip delay Time (RTT) of a communication link of the respective device.
37. The method according to claim 1 , wherein the random selecting uses one or more random numbers generated by a random number generator.
38. The method according to claim 37 , wherein the random number generator is hardware based that uses thermal noise, shot noise, nuclear decaying radiation, photoelectric effect, or quantum phenomena.
39. The method according to claim 37 , wherein the random number generator is software based, and wherein the random number generator is based on executing an algorithm for generating pseudo-random numbers.
40. The method according to claim 1 , wherein the selecting of the first, second, or third IP address, is based on, or is in response to, a time of an action or an event.
41. The method according to claim 40 , wherein the action comprises an action by the first server or by the device addressed by the respectively selected IP address, or wherein the event is an event affecting, or sensed by, the first server or the device addressed by the respectively selected IP address.
42. The method according to claim 1 , further for use with a virtualization, wherein at least one of steps is executed as part of a virtualized application as part of a Virtual Machine (VM), or wherein the first server or any part thereof, is implemented as virtual hardware.
43. The method according to claim 42 , for use with a host computer that implement the VM, wherein the method further comprising executing, by the host computer, a hypervisor or a Virtual Machine Monitor (VMM), and wherein the virtualized application or hardware uses or interfaces virtual hardware.
44. The method according to claim 42 , wherein the virtualization includes, is based on, or uses, full virtualization, para-virtualization, or hardware assisted virtualization.
45. The method according to claim 42 , wherein the first server is implemented as virtual hardware.
46. The method according to claim 45 , for use with a host computer that implement the VM, wherein at least two devices are virtualized by the same host computer.