System and method for URL fetching retry mechanism
A method for overcoming intermittent, temporary, or other fetching failures by using multiple attempts for retrieving a content from a web server to a client device is disclosed. The URL fetching may use direct or non-direct fetching schemes, or a combination thereof. The non-direct fetching method may use intermediate devices, such as proxy server, Data-Center proxy server, tunnel devices, or any combination thereof. Upon sensing a failure of a fetching action, the action is repeated using the same or different parameters or attributes, such as by using different intermediate devices, selected based on different parameters or attributes, such as different countries. The repetitions are limited to a pre-defined maximum number or attempts. The fetching attempts may be performed by the client device, by an intermediate device in a non-direct fetching scheme, or a combination thereof. Various fetching schemes may be used sequentially until the content is retrieved.
1. A method for fetching by a first device over the Internet a content that is stored only in a web server and is identified by a Uniform Resource Locator (URL), for use with a first and second groups of Internet Protocol (IP) addresses, the method comprising:
first fetching, by the first device from the web server, a first response, the first fetching comprising:
selecting, by the first device, a first IP address from the first group;
sending, by the first device, a URL request to the web server using the selected first IP address; and
receiving, by the first device, the first response;
checking the first response for determining whether the first response is a proper response that comprises the content;
responsive to determining that the first response is a proper response, using the fetched content; and
responsive to determining that the first response is not a proper response, performing a second fetching, by the first device from the web server, of a second response, the second fetching comprising:
selecting, by the first device, a second IP address from the second group;
sending, by the first device, an additional URL request to the web server using the selected second IP address;
receiving, by the first device, the second response;
checking the second response for determining whether the second response is a proper response that comprises the content;
responsive to determining that the first response is a proper response, using the fetched content; and
responsive to determining that the second response is not a proper response, performing, by the first device, a first action.
2. The method according to claim 1 , further comprising, in response to the sending of the URL request using the selected first IP address:
receiving, by the web server, the URL request; and
sending, by the web server, the content.
3. The method according to claim 2 , wherein the selected first IP address is used as a source address as part of the URL request, so that the IP address of the first device is unknown to the web server.
4. The method according to claim 1 , wherein each of the IP addresses in the second group is associated with a device, and wherein the sending of the additional URL request to the web server using the selected second IP address comprises:
sending, by the first device, the additional URL request to the device associated with the selected second IP address;
receiving, by the device associated with the selected second IP address, the additional URL request;
sending, by the device associated with the selected second IP address to the web server, the additional URL request;
receiving, by the web server, the URL request;
sending, by the web server to the device associated with the selected second IP address to the web server, the second response; and
sending, by the device associated with the selected second IP address to the first device, the second response.
5. The method according to claim 4 , wherein the selected second IP address is used as a source address as part of the additional URL request, so that the IP address of the first device is unknown to the web server.
6. The method according to claim 1 , further comprising a third fetching, that is a direct fetching scheme, by the first device from the web server, a third response, the third fetching comprising:
sending, by the first device, a URL request to the web server using the first device IP address as a source address; and
receiving, by the first device, the third response from the web server;
checking the third response for determining whether the third response is a proper response that comprises the content; and
responsive to determining that the third response is a proper response, using the fetched content.
7. The method according to claim 5 , further comprising responsive to determining that the third response is not a proper response, performing the first fetching.
8. The method according to claim 5 , wherein the first action comprises the third fetching.
9. The method according to claim 1 , further comprising receiving, by the first device from a server device, the URL request.
10. The method according to claim 1 , further comprising receiving, by the first device from a client device, the URL request.
11. The method according to claim 1 , wherein the first action comprises a third fetching, by the first device, of a third response by using a third fetching scheme that is distinct from the first and second fetchings.
12. The method according to claim 1 , wherein the content comprises, or consists of, a web-page or a web-site that includes, consists of, or comprises, a part or whole of files, text, numbers, audio, voice, multimedia, video, images, music, or computer program.
13. The method according to claim 1 , wherein the first fetching is a proxy-server based fetching scheme, a Data-Center (DC) proxy-server based fetching scheme, or a tunnel-based fetching scheme.
14. The method according to claim 13 , wherein the first device is a server device that consists of, includes, is part of, or is integrated with, a proxy server, that consists of, includes, is part of, or is integrated with, an HTTP proxy server, a web-proxy server, a caching proxy, an open-source caching proxy server, a cloud-based proxy server, an open proxy server, a forwarding proxy server, a reverse proxy server, a transparent proxy server, a non-transparent proxy server, an anonymous proxy server, a translation proxy server, a SOCKS proxy server, a CGI web proxy server, a suffix proxy server, an I2P anonymous proxy server, a DNS proxy server, or any combination thereof.
15. The method according to claim 1 , wherein the first fetching consists of, or comprises, a Data Center (DC)-based proxy server fetching scheme.
16. The method according to claim 1 , wherein the selecting of at least one of the first and second IP addresses is based on, or uses, random selection.
17. The method according to claim 1 , wherein the selecting of at least one of the first and second IP addresses is based on, or uses, an estimated geographical location of the first device or of the web server.
18. The method according to claim 17 , further comprising estimating the geographical location of the first device or of the web server using geolocation that is based on IP geolocation that is based on W3C Geolocation Application Programming Interface (API).
19. The method according to claim 17 , wherein the selecting of at least one of the first and second IP addresses is based on estimated as being in the same area as the first device or the web server.
20. The method according to claim 19 , wherein selecting of at least one of the first and second IP addresses is based on estimated as being in the same continent, country, state, region, city, postal/zip code, latitude, longitude, or Timezone as the first device or the web server.
21. The method according to claim 14 , wherein selecting of at least one of the first and second IP addresses is based on being a recent one to be selected, or based on being the least recent to be selected.
22. The method according to claim 1 , wherein the selecting of at least one of the first and second IP addresses comprises manually selecting by a user.
23. The method according to claim 1 , wherein the checking of at least one of the first and second responses comprises identifying and checking a HTTP status code that is received in response to a sending of the URL request as part of the respective first and second fetchings.
24. The method according to claim 23 , wherein the respective response is determined as a proper response responsive to a status code of 2xx.
25. The method according to claim 24 , wherein the respective response is determined as not being a proper response responsive to a status code of 4xx or 5xx.
26. The method according to claim 25 , wherein the respective response is determined as not being a proper response responsive to a status code of HTTP 404 error message.
27. The method according to claim 23 , wherein the checking of the respective response comprises using a timeout mechanism.
28. The method according to claim 27 , wherein the response is determined as not being a proper response in response to not receiving a proper response after elapsed defined time period after an initiation of the fetching.
29. The method according to claim 23 , wherein the checking if an URL redirection is identified, and wherein the respective response is determined as not being a proper response in response to detecting the URL redirection, and wherein the URL redirection is identified by checking that the HTTP status code is 3xx Redirection.
30. The method according to claim 23 , for use with a criterion, wherein the checking of the response comprises verifying if the content received satisfy the criterion, and wherein the criterion relates to a feature, characteristic, or type, of the received content.
31. The method according to claim 1 , further for use with a virtualization, wherein the first device consists of, comprises, is part of, or is integrated with, a server device that virtualizes a client device that is addressed by the selected first IP address.
32. The method according to claim 31 , wherein the client device virtualization executed as part of a Virtual Machine (VM).
33. The method according to claim 32 , for use with a host computer that implements the VM, wherein the method further comprising executing, by the host computer, a hypervisor or a Virtual Machine Monitor (VMM).
34. The method according to claim 32 , wherein the virtualization includes, is based on, or uses, full virtualization, para-virtualization, or hardware assisted virtualization.