System and method for improving content fetching by selecting tunnel devices
A method for fetching a content from a web server to a client device is disclosed, using tunnel devices serving as intermediate devices. The tunnel device is selected based on an attribute, such as IP Geolocation. A tunnel bank server stores a list of available tunnels that may be used, associated with values of various attribute types. The tunnel devices initiate communication with the tunnel bank server, and stays connected to it, for allowing a communication session initiated by the tunnel bank server. Upon receiving a request from a client to a content and for specific attribute types and values, a tunnel is selected by the tunnel bank server, and is used as a tunnel for retrieving the required content from the web server, using standard protocol such as SOCKS, WebSocket or HTTP Proxy. The client only communicates with a super proxy server that manages the content fetching scheme.
1. A method for fetching a first content identified by a first content identifier and stored in a first web server and a second content identified by a second content identifier and stored in a second web server, for use with first and second servers and first and second intermediate devices that are each connected to the Internet, the method comprising:
identifying the first and second content identifiers using Secure Sockets Layer (SSL) sniffing;
receiving, by the first server from the first client device, a first message that comprises the first content identifier;
sending, by the first server to the second server that is selected from a plurality of servers and that stores the Internet Protocol (IP) addresses of the first and second intermediate devices and their associated geographical locations, a second message that comprises the first geographical location in response to the received first message;
receiving, by the first server from the first intermediate device that is located in a first geographical location, the first content, in response to the sending of the second message;
sending, by the first server to the first client device, the received first content, in response to the receiving of the first content;
receiving, by the first server from the first client device, a third message that comprises the second content identifier;
sending, by the first server to the second server, a fourth message that comprises the second geographical location in response to the received third request message;
receiving, by the first server from the second intermediate device that is located in a second geographical location, the second content, in response to the sending of the fourth message;
sending, by the first server to the first client device, the received second content, in response to the receiving of the second content;
receiving, by the first server from the second client device, a fifth message that comprises the second content identifier;
sending, by the first server to the second server, a sixth message that comprises the second geographical location in response to the received fifth request message;
receiving, by the first server from the second intermediate device, the second content, in response to the sending of the sixth message; and
sending, by the first server to the second client device, the received second content, in response to the receiving of the second content
wherein the second server is selected from a plurality of servers randomly using one or more random numbers generated by a random number generator, or is selected using an attribute value associated with the second server.
2. The method according to claim 1 , wherein further comprising operating, by the first server, a server operating system.
3. The method according to claim 1 , further comprising, in response to the communication initiated by the respective intermediate device, sending, to the respective intermediate device, the first or second content identifier.
4. The method according to claim 1 , wherein the communication over the Internet with the first or second intermediate device, with the second server, or with the first or second client device, is compatible with, a Transmission Control Protocol over Internet Protocol (TCP/IP) protocol or connection.
5. The method according to claim 1 , wherein the communication over the Internet with the first or second intermediate device, with the second server, or with the first or second client device, is compatible with, a Hypertext Transfer Protocol (HTTP) or HTTP Secure (HTTPS) protocol or connection, wherein the first server serves as an HTTP or HTTPS server respectively and the respective intermediate device, the second server, or the respective client device, serves as an HTTP or HTTPS client respectively.
6. The method according to claim 1 , wherein the communication over the Internet with the first or second intermediate device, with the second server, or with the first or second client device, uses, or is compatible with, HTTP Proxy protocol or connection, wherein the first server serves as an HTTP Proxy server and respectively the respective intermediate device, the second server, or the respective client device serves as an HTTP Proxy client.
7. The method according to claim 1 , further comprising establishing a connection with the first or second intermediate device, wherein the first server initiates communication with the intermediate device using the established connection, and wherein the established connection is a TCP connection using ‘Active OPEN’, ‘Passive OPEN’, or TCP keepalive mechanism, or wherein the established connection uses, or is based on, Virtual Private Network (VPN).
8. The method according to claim 1 , further comprising sending, to the first or second client device, the IP address of the first or second intermediate device.
9. The method according to claim 1 , wherein the random number generator is hardware based that uses thermal noise, shot noise, nuclear decaying radiation, photoelectric effect, or quantum phenomena.
10. The method according to claim 1 , for use with a Domain Name System (DNS) server, wherein the first or second content identifier comprises a domain name, the method further comprising performing, using the DNS server, a DNS resolution for obtaining a numerical IP address, and wherein the request message comprises the domain name, and the first message comprises the obtained numerical IP address.
11. The method according to claim 1 , wherein the first or second content comprises a web-page or a web-site, wherein the first or second content identifier is a Uniform Resource Identifier (URI) or Uniform Resource Locator (URL), wherein each of the IP addresses is in IPv4 or IPv6 form, wherein the communication over the Internet with the first or second client device uses, or is compatible with, HTTPS protocol or connection, and wherein the first message is according to, or uses, HTTPS frame or packet form.
12. The method according to claim 1 , wherein the first message comprises an attribute value corresponding to an attribute type, and wherein the method further comprising extracting the attribute value using SSL sniffing.
13. The method according to claim 1 , wherein the first or second geographical location comprises a country, a region, a city, a street, a ZIP code, or a time zone.
14. The method according to claim 1 , wherein the initiated communication by the respective intermediate device uses, or is based on, Network Address Translator (NAT) traversal scheme.
15. The method according to claim 14 , wherein the NAT traversal scheme is according to, or uses, Internet Engineering Task Force (IETF) Request for Comments (RFC) 2663, IETF RFC 3715, IETF RFC 3947, IETF RFC 5128, IETF RFC 5245, IETF RFC 5389, or IETF RFC 7350.
16. The method according to claim 14 , wherein the NAT traversal scheme is according to, or uses, a Traversal Using Relays around NAT (TURN), Socket Secure (SOCKS), NAT ‘hole punching’, Session Traversal Utilities for NAT (STUN), Interactive Connectivity Establishment, (ICE), UPnP Internet Gateway Device Protocol (IGDP), or Application-Level Gateway (ALG).
17. The method according to claim 1 , wherein the communication over the Internet with the first or second intermediate device, with the second server, or with the first or second client device, uses, or is compatible with, a Socket Secure (SOCKS) protocol or connection, wherein the first server serves as an SOCKS server and respectively the intermediate device, the second server, or the client device serves as an SOCKS client.
18. The method according to claim 17 , wherein the SOCKS protocol or connection is according to, or is compatible with, SOCKS4, SOCKS4a, or SOCKS5, and wherein the SOCKS protocol or connection is according to, or is compatible with, IETF RFC 1928, IETF RFC 1929, IETF RFC 1961, or IETF RFC 3089.
19. The method according to claim 1 , for use with a first IP address stored in the first or second client device, wherein the first message comprises the first IP address.
20. The method according to claim 19 , wherein the second message comprises the first IP address.
21. The method according to claim 1 , further comprising selecting the second server from the plurality of servers.
22. The method according to claim 21 , wherein the second server is randomly selected from the plurality of servers.
23. The method according to claim 1 , wherein the random number generator is software based.
24. The method according to claim 23 , wherein the random number generator is based on executing an algorithm for generating pseudo-random numbers.
25. The method according to claim 21 , wherein each of the plurality of servers is associated with an attribute value relating to an attribute type, and wherein the second server is selected from the plurality of servers based on, or according to, the respective attribute value.
26. The method according to claim 25 , wherein the request message comprises the attribute value.
27. The method according to claim 25 , wherein the attribute type is a geographical location, and wherein the attribute value comprises a name or an identifier of a continent, a country, a region, a city, a street, a ZIP code, or a time zone.
28. The method according to claim 27 , wherein the attribute value is based on an actual geographical location or on an IP geolocation.
29. The method according to claim 28 , wherein the IP geolocation is based on W3C Geolocation API.