System and method for improving content fetching by selecting tunnel devices
A method for fetching a content from a web server to a client device is disclosed, using tunnel devices serving as intermediate devices. The tunnel device is selected based on an attribute, such as IP Geolocation. A tunnel bank server stores a list of available tunnels that may be used, associated with values of various attribute types. The tunnel devices initiate communication with the tunnel bank server, and stays connected to it, for allowing a communication session initiated by the tunnel bank server. Upon receiving a request from a client to a content and for specific attribute types and values, a tunnel is selected by the tunnel bank server, and is used as a tunnel for retrieving the required content from the web server, using standard protocol such as SOCKS, WebSocket or HTTP Proxy. The client only communicates with a super proxy server that manages the content fetching scheme.
1. A method for fetching a content that is identified by a content identifier by a client device from a web server using a group of client devices, the method, comprising:
storing, at a first server, a list of client devices by IP addresses associated with the client devices in the group;
establishing a connection with each one of the client devices in the group;
receiving, by a second server from the client device, a request message for the content;
sending, by the second server to a first server, a first message that comprises the content identifier in response to the received request message;
receiving, by the second server from the first server, in response to the sent first message, an IP address of a selected client device from the group that is in an idle state;
sending, by the second server to the selected client device over the established connection, using the received IP address, the content identifier;
receiving, by the second server from a selected client device from the group, the content, in response to the sent content identifier;
sending, by the second server to the client device, the received content in response to the receiving the content from the selected client device;
estimating or measuring, by the second server, a value of a communication property associated with the sending of the content identifier to the selected client device and with the receiving of the content from the selected client device;
sending, by the second server to the first server, the estimated or measured value;
receiving, by the first server from the second server, the sent estimated or measured value; and
updating, by the first server, the communication property value associated with the selected client device, in response to the received estimated or measured value,
wherein the first and second servers and the group of client devices that are each connected to the Internet and are each addressable in the Internet using a respective Internet Protocol (IP) address,
wherein at least one first client device in the group is operating in multiple states that includes the idle state and non-idle states, and
for use with a database that associates a communication property values to each of the client devices in the group.
2. The method according to claim 1 , wherein the estimating or measuring comprises estimating or measuring according to, or based on, IETF RFC 2544 or ITU-T Y.1564 standard.
3. The method according to claim 1 , further comprising:
receiving, by the first server from the second server, the first message; and
selecting, by the first server, the selected client device from the list of client devices by selecting an IP address that is associated with the selected client device, in response to the received first message,
wherein the selecting is based on, or uses, the respective communication property value associated with the selected client device.
4. The method according to claim 1 , further comprising:
sending, by the selected client device to the web server, the content identifier, in response to the receiving of the content identifier by the selected client device; and
receiving, by the selected client device from the web server, the content, in response to the sending of the content identifier by the selected client device.
5. The method according to claim 1 , further comprising:
sending, by the second server to the client device, the IP address of the selected client device;
receiving, by the client device from the second server, the IP address of the selected client device; and
storing, by the client device, the received IP address of the selected client device.
6. The method according to claim 1 , wherein the estimating or measuring comprises estimating or measuring of a delay or time interval from the sending of the content identifier to the selected client device until the receiving of the content from the selected client device.
7. The method according to claim 6 , wherein the communication property comprises, or consists of, Bandwidth (BW) or Round-Trip delay Time (RTT).
8. The method according to claim 1 , wherein the estimating or measuring comprises estimating or measuring of the communication errors in the communication with the selected client device.
9. The method according to claim 8 , wherein the communication property comprises, or consists of, Bit Error Rate (BER), Received Signal Strength Indicator (RSSI), Packet Loss Ratio (PLR), or Cyclic Redundancy Check (CRC) errors.
10. The method according to claim 1 , for use with a first device that is connected to the Internet and addressable in the Internet using a first IP address, the method further comprising:
sending, by the first device to the first server, a third message;
receiving, by the first server from the first device, the third message; and
adding, by the first server the first device to the group of client devices by adding and storing, in the first server, the first IP address to the list, so that the first device can be selected as a client device as part of the selecting by the first server.
11. The method according to claim 10 , wherein the third message comprises at least one value relating to at least one attribute type associated with the first device.
12. The method according to claim 10 , further comprising storing, in the first server, the at least one value, and associating the stored at least one value with the first device or with the first IP address.
13. The method according to claim 10 , further comprising establishing a connection between the first server and the first device, wherein the first server initiates communication with the first device using the established connection.
14. The method according to claim 13 , wherein the established connection is a Transmission Control Protocol (TCP) connection using ‘Active OPEN’, ‘Passive OPEN’, or TCP keepalive mechanism.
15. The method according to claim 13 , wherein the established connection uses, or is based on, Virtual Private Network (VPN).
16. The method according to claim 1 , for use with a first IP address stored in the client device, wherein the request message comprises the first IP address.
17. The method according to claim 16 , wherein the first message comprises the first IP address.
18. The method according to claim 17 , further comprising selecting of the client device from the list of client devices based on, or in response to, the received first IP address.
19. The method according to claim 18 , wherein the selecting by the first server of the client device comprises selecting a client device having the first IP address.
20. The method according to claim 1 , wherein each one of the client devices in the group is operating in multiple states that includes an idle state and non-idle states, the method further comprising by the first client device:
responsive to being in one of the non-idle states, determining, when an idling condition is met;
responsive to the determination that the idling condition is met, shifting to the idle state;
responsive to being in the idle state, determining when an idling condition is met; and
responsive to the determination that the idling condition is not met, shifting to one of the non-idle states.
21. The method according to claim 20 , further comprising receiving, by the first server from the first client device, a message responsive to the first client device state, wherein the first client device is selected by the first server in response to the first client device state being the idle state.
22. The method according to claim 20 , for use with an additional idling condition, wherein the determining comprises determining if the idling condition and the additional idling condition are met.
23. The method according to claim 20 , wherein the selected first client device comprises a network interface or a network transceiver for communication over a network, the method further comprising metering, by the selected first client device, an amount of data transmitted to, or received from, the network during a time interval, and wherein the idling condition is determined to be met based on, or according to, the metered amount of data being under a threshold level.
24. The method according to claim 20 , further comprising:
sending, by the first client device to the first server, a first status message in response to shifting to the idle state; and
sending, by the first client device to the first server, a second status message in response to shifting to a non-idle state.
25. The method according to claim 24 , wherein the first client device is selected by the first server in response to the first or second status message.
26. The method according to claim 24 , further comprising:
receiving, by the first server from the first client device, the first status message; and
adding, the IP address of the first client device to the list of IP addresses in response to received first status message.
27. The method according to claim 26 , further comprising:
receiving, by the first server from the first client device, the second status message; and
removing, the IP address of the first client device from the list of IP addresses in response to received second status message.
28. The method according to claim 20 , further comprising operating, by the first client device, an operating system or a program process or thread, wherein the idling condition is determined to be met based on, or according to, activating or executing the process or thread by the operating system or the program.
29. The method according to claim 28 , wherein the process or thread comprises a low-priority or background task, an idle process, or a screensaver.
30. The method according to claim 28 , wherein the process or thread comprises using an entire screen for displaying.
31. The method according to claim 20 , further comprising monitoring or metering, by the selected first client device, a resource utilization, wherein the idling condition is determined to be met based on, or according to, the monitored or metered resource utilization being under a threshold.
32. The method according to claim 31 , wherein the resource utilization comprises the utilization of a processor in the selected first client device.
33. The method according to claim 20 , wherein the selected first client device comprises an input device for obtaining an input from a human user or operator, the method further comprising sensing, by the selected first client device using the input device, the input, and wherein the idling condition is determined to be met based on, or according to, not receiving an input from the input device for a pre-set time interval.
34. The method according to claim 33 , wherein the input device comprises a pointing device, a keyboard, a touchscreen, or a microphone.
35. The method according to claim 20 , wherein the first client device comprises a motion sensor for sensing motion, acceleration, vibration, or location change of the selected first client device, the method further comprising sensing, by the first client device using the motion sensor, the respective first client device motion, acceleration, vibration, or location change, and wherein the idling condition is determined to be met based on, or according to, respectively sensing the motion, the vibration, the acceleration, or the location change being under a threshold.
36. The method according to claim 35 , wherein the motion sensor comprises an accelerometer, gyroscope, vibration sensor, or a Global Positioning System (GPS) receiver.
37. The method according to claim 20 , wherein the selected first client device comprises a battery, the method further comprising metering or sensing, by the selected first client device, a battery charging level, and wherein the idling condition is determined to be met based on, or according to, the metered or sensed charge level being over a threshold level.
38. The method according to claim 37 , wherein the metering or sensing uses a Battery Management System (BMS).
39. The method according to claim 37 , wherein the threshold level is above 40%, 50%, 60%, 70%, 80%, or 90% of the battery defined full charge capacity.