System and method for URL fetching retry mechanism
A method for overcoming intermittent, temporary, or other fetching failures by using multiple attempts for retrieving a content from a web server to a client device is disclosed. The URL fetching may use direct or non-direct fetching schemes, or a combination thereof. The non-direct fetching method may use intermediate devices, such as proxy server, Data-Center proxy server, tunnel devices, or any combination thereof. Upon sensing a failure of a fetching action, the action is repeated using the same or different parameters or attributes, such as by using different intermediate devices, selected based on different parameters or attributes, such as different countries. The repetitions are limited to a pre-defined maximum number or attempts. The fetching attempts may be performed by the client device, by an intermediate device in a non-direct fetching scheme, or a combination thereof. Various fetching schemes may be used sequentially until the content is retrieved.
1. A method for anonymously fetching by a first device over the Internet a content that is stored in a web server and is identified by a Uniform Resource Locator (URL), for use with a first server that stores a first list of Internet Protocol (IP) addresses, and for use with a second server that stores a second list of IP addresses, the method comprising:
sending, by the first device to the first server, a URL request;
receiving, by the first server from the first device, the URL request;
selecting, by the first server, a first IP address from the first list;
sending, by the first server to the web server using the selected first IP address as a source address, the URL request;
receiving, by the first server from the web server, a first response;
sending, by the first server to the first device, the first response; and
determining whether the first response is a proper response that comprises the content,
in response to determining that the first response is not a proper response by verifying that the received first response does not satisfy a criterion:
sending, by the first device to the second server, the URL request;
receiving, by the second server from the first device, the URL request;
selecting, by the second server, a second IP address from the second list;
sending, by the second server to a second device that is addressed by the selected second IP address, the URL request;
receiving, by the second device from the second server, the URL request;
sending, by the second device to the web server, the URL request;
receiving, by the second device from the web server, a second response;
sending, by the second device to the second server, the second response; and
sending, by the second server to the first device, the second response,
wherein the selecting of the first IP address or the selecting of the second IP address is based on, or uses, load balancing.
2. The method according to claim 1 , wherein the content comprises, or consists of, a web-page or a web-site that includes, consists of, or comprises, a part or whole of a file, text data, audio data, voice data, multimedia data, video data, image data, music data, or a computer program.
3. The method according to claim 1 , wherein the first device consists of, comprises, is part of, or is integrated with, a server device.
4. The method according to claim 1 , wherein the first server and the second server are the same server device.
5. The method according to claim 1 , further comprising determining whether the second response is a proper response that comprises the content.
6. The method according to claim 5 , further comprising, in response to determining that the first response is not a proper response:
sending, by the first device to the web server, the URL request;
receiving, by the first device from the web server, a third response; and
determining whether the third response is a proper response that comprises the content.
7. The method according to claim 1 , further comprising, in response to determining that the first response is not a proper response:
selecting, by the first server, a third IP address from the first list;
sending, by the first server to the web server using the selected third IP address as the source address, the URL request;
receiving, by the first server from the web server, a third response;
sending, by the first server to the first device, the third response; and
determining whether the third response is a proper response that comprises the content,
wherein the sending, by the first device to the second server, of the URL request is further in response to determining that the third response is not a proper response.
8. The method according to claim 1 , further comprising determining whether the second response is a proper response that comprises the content, and in response to determining that the first response is not a proper response:
selecting, by the second server, a third IP address from the second list;
sending, by the second server to a third device that is addressed by the selected third IP address, the URL request;
receiving, by the third device from the second server, the URL request;
sending, by the third device to the web server, the URL request;
receiving, by the third device from the web server, a third response;
sending, by the third device to the second server, the third response; and
sending, by the second server to the first device, the third response.
9. The method according to claim 1 , wherein the URL request consists of, or is part of, a Hypertext Transfer Protocol (HTTP) request.
10. The method according to claim 1 , wherein the first server or the second server consists of, includes, is part of, or is integrated with, a proxy server.
11. The method according to claim 10 , wherein the proxy server consists of, includes, is part of, or is integrated with, an HTTP proxy server, a web-proxy server, a caching proxy, an open-source caching proxy server, a cloud-based proxy server, an open proxy server, a forwarding proxy server, a reverse proxy server, a transparent proxy server, a non-transparent proxy server, an anonymous proxy server, a translation proxy server, a SOCKS proxy server, a CGI web proxy-5-server, a suffix proxy server, an 2P anonymous proxy server, a DNS proxy server, or any combination thereof.
12. The method according to claim 1 , wherein the selecting of the first IP address or the selecting of the second IP address is based on, or uses, random selecting.
13. The method according to claim 1 , wherein the selecting of the first IP address or the selecting of the second IP address is based on, or uses, random selection.
14. The method according to claim 13 , wherein random selection uses, or is based on, one or more random numbers generated by a random number generator.
15. The method according to claim 14 , wherein the random number generator is hardware based.
16. The method according to claim 15 , wherein the random number generator is using thermal noise, shot noise, nuclear decaying radiation, photoelectric effect, or quantum phenomena.
17. The method according to claim 14 , wherein the random number generator is software based.
18. The method according to claim 17 , wherein the random number generator is based on executing an algorithm for generating pseudo-random numbers.
19. The method according to claim 14 , wherein the selecting of the first IP address or the selecting of the second IP address is based on, or uses, an estimated geographical location of the first device or of the web server.
20. The method according to claim 19 , further comprising estimating a geographical location of the first IP address the second IP address, the first device, or the web server, using geolocation.
21. The method according to claim 20 , wherein the geolocation is based on IP geolocation.
22. The method according to claim 20 , wherein the geolocation is based on W3C Geolocation Application Programming Interface (API).
23. The method according to claim 19 , wherein the selecting of the first IP address or the selecting of the second IP address is based on, or uses, a geographical location that is estimated as being in the same area as the first device or the web server.
24. The method according to claim 23 , wherein the location is estimated as being in the same continent, country, state, region, city, postal/zip code, latitude, longitude, or Timezone as the first device or the web server.
25. The method according to claim 1 , wherein the selecting of the first IP address or the selecting of the second IP address is based on being the recent one to be selected, or based on being the least recent to be selected.
26. The method according to claim 1 , further for use with a virtualization, wherein the first device, the second device, the first server, or the second server consists of, comprises, is part of, or is integrated with, a server device that virtualizes the respective device.
27. The method according to claim 26 , wherein the virtualization executed as part of a Virtual Machine (VM).
28. The method according to claim 27 , for use with an host computer that implements the VM, wherein the method further comprising executing, by the host computer, a hypervisor or a Virtual Machine Monitor (VMM).
29. The method according to claim 27 , wherein the virtualization includes, is based on, or uses, full virtualization, para-virtualization, or hardware assisted virtualization.
30. The method according to claim 1 , wherein the determining comprises identifying and checking a HTTP status code.
31. The method according to claim 30 , wherein the first response is determined as a proper response responsive to a status code of 2xx.
32. The method according to claim 31 , wherein the first response is determined as not being a proper response responsive to a status code of 5xx.
33. The method according to claim 31 , wherein the first response is determined as not being a proper response responsive to a status code of 4xx.
34. The method according to claim 30 , wherein the first response is determined as not being a proper response responsive to a status code of HTTP 404 error message.
35. The method according to claim 30 , wherein the checking of the first response comprises using a timeout mechanism.
36. The method according to claim 35 , wherein the first response is determined as not being a proper response in response to not receiving a proper response after elapsed defined time period after an initiation of the fetching.
37. The method according to claim 30 , wherein the checking comprises identifying an URL redirection, and wherein the first response is determined as not being a proper response in response to detecting the URL redirection.
38. The method according to claim 37 , wherein the URL redirection is identified by checking that the HTTP status code is 3xx Redirection.
39. The method according to claim 1 , wherein the criterion relates to a feature, characteristic, or type, of the received content.
40. The method according to claim 1 , wherein the criterion comprises a value, and wherein the first response is determined as not being a proper response in response to comparing the content feature, characteristic, or type, to the value.
41. The method according to claim 40 , wherein the criterion comprises a value of a size of a file, and wherein the first response is determined as not being a proper response in response to comparing the received content size to the value.
42. A non-transitory computer readable medium containing computer instructions that, when executed by a processor, cause the processor to perform at least part of the steps of claim 1 .
43. A non-transitory computer readable medium containing computer instructions that, when executed by a processor, cause the processor to perform all of the steps of claim 1 .
44. The method according to claim 1 , wherein at least part of steps of claim 1 are included in a Software Development Kit (SDK) that is provided as a non-transitory computer readable medium containing computer instructions, and wherein the method further comprising installing the SDK.
45. The method according to claim 1 , wherein the first device or the second device comprises, or consists of, a client device, and the method further comprising storing, operating, or using, by the client device, a client operating system.
46. The method according to claim 45 , wherein the client operating system consists of, comprises, or is based on, one out of Microsoft Windows 7, Microsoft Windows XP, Microsoft Windows 8, Microsoft Windows 8.1, Linux, and Google Chrome OS.
47. The method according to claim 45 , wherein the client operating system is a Real-Time Operating System (RTOS).
48. The method according to claim 47 , wherein the RTOS comprises FreeRTOS, SafeRTOS, QNX, VxWorks, or Micro-Controller Operating Systems (pC/OS).
49. The method according to claim 45 , further comprising storing, operating, or using, by the client device, a web browser.
50. The method according to claim 49 , wherein the web browser consists of, comprises of, or is based on, Microsoft Internet Explorer, Google Chrome, Opera™, or Mozilla Firefox®.
51. The method according to claim 49 , wherein the web browser is a mobile web browser.
52. The method according to claim 51 , wherein the mobile web browser consists of, comprises of, or is based on, Safari, Opera Mini™, or Android web browser.