System and method for URL fetching retry mechanism
A method for overcoming intermittent, temporary, or other fetching failures by using multiple attempts for retrieving a content from a web server to a client device is disclosed. The URL fetching may use direct or non-direct fetching schemes, or a combination thereof. The non-direct fetching method may use intermediate devices, such as proxy server, Data-Center proxy server, tunnel devices, or any combination thereof. Upon sensing a failure of a fetching action, the action is repeated using the same or different parameters or attributes, such as by using different intermediate devices, selected based on different parameters or attributes, such as different countries. The repetitions are limited to a pre-defined maximum number or attempts. The fetching attempts may be performed by the client device, by an intermediate device in a non-direct fetching scheme, or a combination thereof. Various fetching schemes may be used sequentially until the content is retrieved.
1. A method for use with a content that comprises a web-page or a part thereof and that is stored only in a web server and is identified by a Uniform Resource Locator (URL), for use with a first and second lists of Internet Protocol (IP) addresses, the method comprising:
identifying, by a first device, the URL;
selecting, by the first device, a first IP address from the first list;
sending, by the first device to the web server over the Internet, a first Hypertext Transfer Protocol (HTTP) request that comprises the URL, using the selected first IP address as a source address;
receiving, by the first device over the Internet, a first response in response to the sent first HTTP request;
determining whether the first response is a proper response that comprises the content; and
responsive to determining that the first response is not a proper response:
selecting, by the first device, a second IP address from the second list;
sending, by the first device to the web server over the Internet, a second Hypertext Transfer Protocol (HTTP) request that comprises the URL, using the selected second IP address as a source address;
receiving, by the first device over the Internet, a second response in response to the sent second HTTP request;
determining whether the second response is a proper response that comprises the content;
responsive to determining that the second response is a proper response, using the content received in the second response; and
responsive to determining that the second response is not a proper response, performing, by the first device, a first action.
2. The method according to claim 1 , further comprising responsive to determining that the first response is a proper response, using the content received in the first response.
3. The method according to claim 1 , wherein the selected first IP address is used as a source address so that the IP address of the first device is unknown to the web server as a requesting device.
4. The method according to claim 1 , wherein each of the IP addresses in the second list addresses a respective device, and wherein the sending of the second HTTP request by the first device to the web server comprises:
sending, by the first device over the Internet to a second device that is addresses by the selected second IP address, the second HTTP request;
receiving, by the second device over the Internet, the second URL request sent by the first device;
sending, by the second device to the web server over the Internet, the second HTTP request;
receiving, by the second device from the web server over the Internet, the second response; and
sending, by the second device to the first device over the Internet, the received second response.
5. The method according to claim 1 , wherein the IP address of the first device is unknown to the web server as the requesting device.
6. The method according to claim 1 , further comprising:
sending, by the first device over the Internet to the web server using the first device IP address as a source address, a third HTTP request that comprises the URL;
receiving, by the first device from the web server over the Internet, a third response;
determining whether the third response is a proper response that comprises the content; and
responsive to determining that the third response is a proper response, using the content in the third response.
7. The method according to claim 6 , wherein the selecting, by the first device, of the first IP address from the first list is in response to determining that the third response is not a proper response.
8. The method according to claim 6 , wherein the first action comprises the sending, by the first device over the Internet to the web server, of the third HTTP request.
9. The method according to claim 1 , wherein the identifying of the URL comprises receiving, by the first device from a server device over the Internet, the URL.
10. The method according to claim 1 , wherein the identifying of the URL comprises receiving, by the first device from a client device over the Internet, the URL.
11. The method according to claim 1 , wherein the first action comprises:
selecting, by the first device, a third IP address from the first or second list;
sending, by the first device to the web server over the Internet, a third Hypertext Transfer Protocol (HTTP) request that comprises the URL, using the selected third IP address as the source address;
receiving, by the first device over the Internet, a third response in response to the sent third HTTP request; and
determining whether the third response is a proper response that comprises the content.
12. The method according to claim 1 , wherein the content comprises, or consists of, a computer file, audio data, voice data, multimedia data, video data, an image, music, or a computer program.
13. The method according to claim 1 , wherein the first device comprises a proxy server or a Data-Center (DC) proxy-server.
14. The method according to claim 13 , wherein the proxy server consists of, includes, is part of, or is integrated with, an HTTP proxy server, a web-proxy server, a caching proxy, an open-source caching proxy server, a cloud-based proxy server, an open proxy server, a forwarding proxy server, a reverse proxy server, a transparent proxy server, a non-transparent proxy server, an anonymous proxy server, a translation proxy server, a SOCKS proxy server, a CGI web proxy server, a suffix proxy server, an I2P anonymous proxy server, a DNS proxy server, or any combination thereof.
15. The method according to claim 1 , wherein the first device is a client device.
16. The method according to claim 1 , wherein the selecting of at least one of the first and second IP addresses is based on, or uses, random selection.
17. The method according to claim 1 , wherein the selecting of at least one of the first and second IP addresses is based on, or uses, an estimated geographical location of the first device or of the web server.
18. The method according to claim 17 , further comprising estimating a geographical location of the first device or of the web server using geolocation, that is based on IP geolocation that uses W3C Geolocation Application Programming Interface (API).
19. The method according to claim 17 , wherein the selecting of at least one of the first and second IP addresses is based on estimated as being in a same region as the first device or the web server.
20. The method according to claim 19 , wherein the region comprises a country, state, region, city, postal/zip code, latitude, longitude, or Timezone.
21. The method according to claim 1 , wherein selecting of at least one of the first and second IP addresses is based on being a respective recent one to be selected, or based on being the least respective recent to be selected.
22. The method according to claim 1 , wherein the selecting of at least one of the first and second IP addresses comprises manually selecting by a user.
23. The method according to claim 1 , wherein the first response is a first HTTP response, and wherein the determining comprises determining that the first response is a proper response comprises identifying and checking a HTTP status code in the first HTTP response, or wherein the second response is a second HTTP response, and wherein the determining comprises determining that the second response is a proper response comprises identifying and checking a HTTP status code in the second HTTP response.
24. The method according to claim 23 , wherein the respective response is determined as a proper response responsive to a status code of 2xx.
25. The method according to claim 24 , wherein the respective response is determined as not being a proper response responsive to a status code of 4xx or 5xx.
26. The method according to claim 25 , wherein the respective response is determined as not being a proper response responsive to a status code of HTTP 404 error message.
27. The method according to claim 23 , wherein the determining of the respective response comprises using a timeout mechanism.
28. The method according to claim 27 , wherein the response is determined as not being a proper response in response to not receiving a proper response after elapsed defined time period after the sending of the respective HTTP request.
29. The method according to claim 23 , wherein the determining comprises determining if an URL redirection is identified, and wherein the respective response is determined as not being a proper response in response to detecting the URL redirection, and wherein the URL redirection is identified by checking that the HTTP status code is 3xx Redirection.
30. The method according to claim 23 , for use with a criterion, wherein the determining of the respective response comprises verifying if the content received in the respective response satisfies the criterion, and wherein the criterion relates to a feature, characteristic, or type, of the received content in the respective response.
31. The method according to claim 1 , further for use with a virtualization, wherein the first device consists of, comprises, is part of, or is integrated with, a server device that virtualizes a client device that is addressed by the selected first IP address.
32. The method according to claim 31 , wherein the client device virtualization executed as part of a Virtual Machine (VM).
33. The method according to claim 32 , for use with a host computer that implements the VM, wherein the method further comprising executing, by the host computer, a hypervisor or a Virtual Machine Monitor (VMM).
34. The method according to claim 32 , wherein the virtualization includes, is based on, or uses, a full virtualization, para-virtualization, or hardware assisted virtualization.