System and method for URL fetching retry mechanism
A method for overcoming intermittent, temporary, or other fetching failures by using multiple attempts for retrieving a content from a web server to a client device is disclosed. The URL fetching may use direct or non-direct fetching schemes, or a combination thereof. The non-direct fetching method may use intermediate devices, such as proxy server, Data-Center proxy server, tunnel devices, or any combination thereof. Upon sensing a failure of a fetching action, the action is repeated using the same or different parameters or attributes, such as by using different intermediate devices, selected based on different parameters or attributes, such as different countries. The repetitions are limited to a pre-defined maximum number or attempts. The fetching attempts may be performed by the client device, by an intermediate device in a non-direct fetching scheme, or a combination thereof. Various fetching schemes may be used sequentially until the content is retrieved.
1 . A method for fetching, using a group of client devices, a content that comprises a web-page or a part thereof and is stored in a web server and is identified by a Uniform Resource Locator (URL), the method comprising:
identifying, the URL;
repetitively fetching a response from the web server until the earlier of fetching N times, wherein N>1, and the response determined to be a proper response, wherein each one of the fetchings comprising:
selecting a client device from the group;
sending, to the respective selected client device over the Internet, a Hypertext Transfer Protocol (HTTP) request that comprises the URL;
receiving, from the respective selected client device over the Internet, a respective HTTP response in response to the respective sending of the HTTP request; and
determining whether the respective HTTP response is a proper response that comprises the content by checking a HTTP status code of the HTTP response,
wherein responsive to determining that the each one of the responses is not a proper response, sending an error message,
wherein a distinct client device is selected in each one of the fetchings, and
wherein N is equal to, or more than, 2, 3, 4, or 5.
2 . The method according to claim 1 , wherein N is equal to, or more than, 7, 10, 15, 20, 25, 30, 40, 50, or 100.
3 . The method according to claim 1 , further being performed by a requesting client device.
4 . The method according to claim 1 , further comprising storing and operating a web browser.
5 . The method according to claim 4 , wherein the identifying of the URL is performed by the web browser, or wherein the sending of the error message comprises returning the error message to the web browser.
6 . The method according to claim 4 , wherein at least part of the steps are performed as part of an operation of a web browser.
7 . The method according to claim 1 , further comprising storing, operating, or using, a web browser, that consists of, comprises of, or is based on, Microsoft Internet Explorer, Google Chrome, Opera™, or Mozilla Firefox®, or wherein the web browser is a mobile web browser that consists of, comprises of, or is based on, Safari, Opera Mini™, or Android web browser.
8 . The method according to claim 1 , wherein any HTTP request comprises a Hypertext Transfer Protocol Secure (HTTPS) request, and wherein any HTTP response comprises a Hypertext Transfer Protocol Secure (HTTPS) response.
9 . The method according to claim 1 , wherein a communication with at least one of the selected client devices is based on, uses, or is compatible with, Hypertext Transfer Protocol (HTTP) or Hypertext Transfer Protocol Secure (HTTPS) protocol or connection.
10 . The method according to claim 1 , wherein the content includes, consists of, or comprises, a computer file, audio data, voice data, multimedia data, video data, an image, or a computer program.
11 . The method according to claim 1 , wherein any HTTP response is determined as a proper response responsive to a status code of 2xx.
12 . The method according to claim 1 , wherein any response is determined as not being a proper response HTTP responsive to a status code of 4xx or 5xx.
13 . The method according to claim 1 , further comprising, by one of, or each of, the selected client devices receiving, by the selected device the respective HTTP request, in response to the sending of the HTTP request, and in response to the receiving of the respective HTTP request, repetitively fetching a response from the web server until the earlier of fetching M times, wherein M=1 or M>1, and the response determined to be a proper response, wherein each one of the fetchings by the one of, or each of, the selected client devices, comprising:
fetching M times, wherein M>1 or M=1, by the respective selected client device from the web server, a second response, until the second response is determined to be a proper response, wherein each one of the M times fetchings comprises:
sending, by the respectively selected client device to the web server over the Internet, a respective first HTTP request that comprises the URL;
receiving, by the respectively selected client device from the web server over the Internet, a respective first HTTP response in response to the sending of the respective first HTTP request; and
determining whether the respective first HTTP response is a proper response that comprises the content,
wherein responsive to determining that any one of the respective first HTTP response is a proper response, sending, by the respective selected client device, the content in the proper response, and
wherein responsive to determining that each of the respective first HTTP responses is not a proper response, sending, by the respective selected client device, an additional error message.
14 . The method according to claim 13 , wherein M is equal to, or more than, 2, 3, 4, 5, 7, 10, 15, 20, 25, 30, 40, 50, or 100.
15 . The method according to claim 13 , wherein M is less than 2, 3, 4, 5, 7, 10, 15, 20, 25, 30, 40, 50, 100, 150, or 200.
16 . The method according to claim 1 , wherein any HTTP response is determined as not being a proper response responsive to a status code of HTTP 404 error message.
17 . The method according to claim 1 , wherein any HTTP response is determined as not being a proper response in response to not receiving a proper response after elapsed defined time period after an initiation of the fetching using a timeout mechanism.
18 . The method according to claim 1 , wherein any HTTP response is determined as not being a proper response in response to detecting a URL redirection.
19 . The method according to claim 18 , wherein the URL redirection is determined by checking that the HTTP status code in the received HTTP response is 3xx Redirection.
20 . The method according to claim 1 , for use with a criterion, wherein any HTTP response is determined as proper further in response to satisfying the criterion.
21 . The method according to claim 20 , wherein the criterion is associated with a feature, characteristic, or type, of the received respective HTTP response, or wherein the criterion comprises a value, and the respective HTTP response is determined as not being a proper response in response to comparing to the value.
22 . The method according to claim 1 , for use with a list that comprises Internet Protocol (IP) addresses of the client devices in the group, wherein any selecting of any client device from the group comprises, or consists of, selecting the respective IP address of the selected client device from the list.
23 . The method according to claim 1 , wherein at least one of, or each one of, the client devices in the group is housed in a hand-held enclosure or a portable enclosure.
24 . The method according to claim 1 , wherein at least one of, or each one of, the client devices in the group is integrated in part or entirely in an appliance.
25 . The method according to claim 1 , wherein at least one of, or each one of, the client devices in the group consists of, comprises, is integrated with, or is part of, a wearable device that is wearable on a person.
26 . The method according to claim 1 , further comprising storing, operating, or using an operating system or a mobile operating system by at least one of, or each one of, the client devices in the group.
27 . The method according to claim 1 , wherein at least one of, or each one of, the selecting comprises random selecting.
28 . The method according to claim 1 , wherein at least one of, or each one of, the selecting is based on, or uses, an estimated geographical location of the selected client device.
29 . The method according to claim 28 , further comprising estimating the geographical location of the selected client device using geolocation.
30 . The method according to claim 29 , wherein the geolocation comprises an IP geolocation that is based on W3C Geolocation Application Programming Interface (API).
31 . The method according to claim 28 , wherein a selected client device is selected based on estimated as being associated to being in the same area as the web server.
32 . The method according to claim 1 , further comprising storing, operating, or using, by one or more of the client devices in the group, a client operating system.
33 . The method according to claim 32 , wherein client operating system consists of, comprises, or is based on, Microsoft Windows 7, Microsoft Windows XP, Microsoft Windows 8, Microsoft Windows 8.1, Linux, or Google Chrome OS.
34 . The method according to claim 1 , wherein at least one of the steps is performed by a server device.
35 . The method according to claim 34 , wherein the server device consists of, includes, is part of, or is integrated with, a proxy server.
36 . The method according to claim 35 , wherein the proxy server consists of, includes, is part of, or is integrated with, an HTTP proxy server, a web-proxy server, a caching proxy, an open-source caching proxy server, a cloud-based proxy server, an open proxy server, a forwarding proxy server, a reverse proxy server, a transparent proxy server, a non-transparent proxy server, an anonymous proxy server, a translation proxy server, a Socks proxy server, a CGI web proxy server, a suffix proxy server, an I2P anonymous proxy server, a DNS proxy server, or any combination thereof.
37 . The method according to claim 1 , wherein at least one of the steps is performed as part of a plug-in or an extension integrated with a web browser.
38 . The method according to claim 37 , wherein at least one of the steps is performed by a plug-in or an extension.
39 . The method according to claim 37 , wherein the identifying of the URL uses a plug-in or an extension to a web browser.
40 . The method according to claim 37 , wherein the integration is hooking to the web browser, wherein the integration is in a filter driver form, or wherein the web browser and the steps are communicating using an Inter-Process Communication (IPC).
41 . The method according to claim 37 , wherein the identifying of the URL uses hooking to the web browser, a filter driver form, or using an Inter-Process Communication (IPC).
42 . The method according to claim 41 , wherein the IPC uses a file sharing, a signal, a socket, a pipe, a message queue, a shared memory, a semaphore, or memory mapped file, or wherein the IPC is using a clipboard, a Component Object Model (COM), a data copy, a DDE protocol, or mailslots.
43 . The method according to claim 1 , wherein the communication over the Internet with any of the client devices in the group is based on, uses, or is compatible with, a Socket Secure (SOCKS) protocol or connection.
44 . The method according to claim 1 , wherein the SOCKS protocol or connection is according to, based on, or is compatible with, SOCKS4, SOCKS4a, or SOCKS5, or wherein the SOCKS protocol or connection is according to, based on, or is compatible with, IETF RFC 1928, IETF RFC 1929, IETF RFC 1961, or IETF RFC 3089.
45 . The method according to claim 1 , wherein the communication over the Internet with one or more of the selected devices is based on, uses, or is compatible with, WebSocket (ws) or WebSocket Secure (wss) protocol or connection.