System and method for URL fetching retry mechanism
A method for overcoming intermittent, temporary, or other fetching failures by using multiple attempts for retrieving a content from a web server to a client device is disclosed. The URL fetching may use direct or non-direct fetching schemes, or a combination thereof. The non-direct fetching method may use intermediate devices, such as proxy server, Data-Center proxy server, tunnel devices, or any combination thereof. Upon sensing a failure of a fetching action, the action is repeated using the same or different parameters or attributes, such as by using different intermediate devices, selected based on different parameters or attributes, such as different countries. The repetitions are limited to a pre-defined maximum number or attempts. The fetching attempts may be performed by the client device, by an intermediate device in a non-direct fetching scheme, or a combination thereof. Various fetching schemes may be used sequentially until the content is retrieved.
1 . A method for obtaining over the Internet by a requesting device, using a group of devices, a web-page that is stored in a web server and is identified by a Uniform Resource Locator (URL), the method comprising:
(a) selecting, by the requesting device, a device from the group;
(b) sending, by the requesting device to the selected device over the Internet, a Hypertext Transfer Protocol (HTTP) request that comprises the URL;
(c) receiving, by the requesting device from the selected device over the Internet, a response in response to the sending of the HTTP request; and
(d) identifying and checking, by the requesting device, a HTTP status code for determining whether the response is a proper response that comprises the web-page,
until the response is determined to be a proper response, repeating steps (a)-(d) N times, wherein N>1, using a different selected device in each repetition,
responsive to determining in all the repetitions that the response is not a proper response, returning or sending an error message, and
wherein N is equal to, or more than, 2, 3, or 4.
2 . The method according to claim 1 , wherein the communication over the Internet between the requesting device and each of the respective selected devices is based on, uses, or is compatible with, Hypertext Transfer Protocol (HTTP) or Hypertext Transfer Protocol Secure (HTTPS) protocol or connection.
3 . The method according to claim 2 , wherein the requesting device serves as an HTTP or HTTPS client.
4 . The method according to claim 2 , wherein each of the respective selected devices serves as an HTTP or HTTPS client.
5 . The method according to claim 1 , wherein the HTTP request comprises, or consists of, an HTTPS request.
6 . The method according to claim 1 , wherein the error message comprises an HTTP message.
7 . The method according to claim 1 , wherein the web-page includes, consists of, or comprises, a part or whole of, a computer file, audio data, voice data, multimedia data, video data, an image, a computer program, or any combination thereof, and wherein N is equal to, or more than 5, 7, 10, 15, 20, 25, 30, 40, 50, or 100.
8 . The method according to claim 1 , wherein the response is determined as a proper response responsive to a status code of 2xx, and wherein the response is determined as not being a proper response responsive to a status code of 4xx or 5xx.
9 . The method according to claim 1 , further comprising, by at least one of, or by each of, the selected devices:
receiving, by the selected device from the requesting device over the Internet, the respective HTTP request, in response to the sending by the requesting device of the respective HTTP request, and responsive to the receiving of the HTTP request by the selected device:
(e) sending, by the selected device to the web server over the Internet, the respective HTTP request;
(f) receiving, by the selected device from the web server over the Internet, a respective response in response to the sending of the HTTP request; and
(g) sending, by the selected device to the requesting device over the Internet, the received respective response.
10 . The method according to claim 9 , further comprising repeating steps (e)-(g) M times, wherein M>1 or M=1.
11 . The method according to claim 10 , further comprising for each repetition step (h) identifying and checking, by the selected device, a HTTP status code for determining whether the response is a proper response that comprises the web-page.
12 . The method according to claim 11 , further comprising until the response is determined to be a proper response, repeating steps (e)-(g) M times, wherein M>1.
13 . The method according to claim 12 , further comprising responsive to determining in all the repetitions that the response is not a proper response, sending, by the selected device to the requesting device over the Internet, an additional error message.
14 . The method according to claim 13 , wherein M is equal to, or more than, 2, 3, 4, 5, 7, 10, 15, 20, 25, 30, 40, 50, or 100, or wherein M is less than 2, 3, 4, 5, 7, 10, 15, 20, 25, 30, 40, 50, 100, 150, or 200.
15 . The method according to claim 9 , further comprising operating, by the selected device, a web browser.
16 . The method according to claim 15 , wherein the sending, by the selected device to the web server over the Internet, of the respective HTTP request, is performed as part of the operating of the web browser.
17 . The method according to claim 1 , wherein the response is determined as not being a proper response responsive to a status code of HTTP 404 error message.
18 . The method according to claim 1 , wherein the checking of the response further comprises using a timeout mechanism, wherein the response is determined as not being a proper response in response to not receiving a proper response after elapsed defined time period after an initiation of the sending.
19 . The method according to claim 1 , wherein the checking comprises checking if an URL redirection is identified, and wherein the response is determined as not being a proper response in response to detecting the URL redirection, and wherein the URL redirection is identified by checking that an HTTP status code received is 3xx Redirection.
20 . The method according to claim 1 , for use with a criterion, wherein the checking of the response further comprises verifying if the response received satisfy the criterion, and wherein the criterion relates to a feature, characteristic, or type, of the received response, or wherein the criterion comprises a value, and the response is determined as not being a proper response in response to comparing the response feature, characteristic, or type, to the value.
21 . The method according to claim 1 , wherein more than 2 repetitions are performed, and wherein the same device from the group is selected for all the repetitions.
22 . The method according to claim 1 , wherein more than 2 repetitions are performed, and wherein different devices from the group are selected in the repetitions.
23 . The method according to claim 1 , for use with a list that comprises Internet Protocol (IP) addresses of the devices in the group, wherein the selecting of a device from the group comprises, or consists of, selecting the respective IP address of the selected device.
24 . The method according to claim 23 , wherein at least one of, or each one of, the devices in the group is a client device that is housed in a single enclosure that is a hand-held enclosure or a portable enclosure.
25 . The method according to claim 23 , wherein at least one of, or each one of, the devices in the group is a client device that is integrated in part or entirely in an appliance.
26 . The method according to claim 23 , wherein at least one of, or each one of, the devices in the group consists of, comprises, is integrated with, or is part of, a wearable device that is wearable on a person.
27 . The method according to claim 23 , wherein at least one of, or each one of, the devices in the group is a client device that is operative for storing, operating, or using an operating system or a mobile operating system.
28 . The method according to claim 1 , wherein the selecting is based on, or uses, random selection.
29 . The method according to claim 1 , wherein the selecting is based on, or uses, an estimated geographical location of the selected device.
30 . The method according to claim 29 , further comprising estimating the geographical location of the selected device.
31 . The method according to claim 30 , wherein the estimating uses geolocation or IP geolocation that is based on W3C Geolocation Application Programming Interface (API).
32 . The method according to claim 29 , wherein a device is selected based on estimated as being associated to being in a same area as the requesting device or as the web server.
33 . The method according to claim 1 , wherein the requesting device is part of, or is integrated in part or entirely, in an appliance.
34 . The method according to claim 1 , wherein the requesting device, consists of, comprises, is integrated with, or is part of, a wearable device that is wearable on a person.
35 . The method according to claim 1 , wherein the requesting device consists of, comprises, is part of, or is integrated with, a client device.
36 . The method according to claim 35 , further comprising storing, operating, or using, by the requesting device or by one or more of the devices in the group, a client operating system.
37 . The method according to claim 36 , wherein the client operating system consists of, comprises, or is based on, Microsoft Windows 7, Microsoft Windows XP, Microsoft Windows 8, Microsoft Windows 8.1, Linux, Google Chrome OS, or any combination thereof.
38 . The method according to claim 1 , wherein the requesting device consists of, comprises, is part of, or is integrated with, a server device.
39 . The method according to claim 1 , wherein at least one of, or each one of, the devices in the group, consists of, comprises, is part of, or is integrated with, a server device.
40 . The method according to claim 38 , wherein the server device consists of, includes, is part of, or is integrated with, a proxy server, wherein the proxy server consists of, includes, is part of, or is integrated with, an HTTP proxy server, a web-proxy server, a caching proxy, an open-source caching proxy server, a cloud-based proxy server, an open proxy server, a forwarding proxy server, a reverse proxy server, a transparent proxy server, a non-transparent proxy server, an anonymous proxy server, a translation proxy server, a Socks proxy server, a CGI web proxy server, a suffix proxy server, an I2P anonymous proxy server, a DNS proxy server, or any combination thereof.
41 . The method according to claim 1 , further comprising operating, by the requesting device, a web browser application.
42 . The method according to claim 41 , further comprising identifying, by the requesting device, the URL as part of the operating of the web browser.
43 . The method according to claim 42 , wherein the error message is sent or returned to the web browser.
44 . The method according to claim 42 , further comprising, responsive to determining that the response is a proper response, using, by the web browser, the response.
45 . The method according to claim 41 , wherein at least one of the steps is performed as part of a plug-in or an extension integrated with the web browser.
46 . The method according to claim 45 , wherein at least one of the steps performed by the requesting device is integrated with the web browser in a form of a plug-in or an extension, or wherein the identifying of the HTTP request uses a plug-in or an extension to the web browser.
47 . The method according to claim 46 , wherein the integration is by hooking to the web browser, wherein the integration is in a filter driver form, or wherein the web browser and the steps are communicating using an Inter-Process Communication (IPC).
48 . The method according to claim 42 , wherein the identifying of the HTTP to the web request uses hooking browser, a filter driver form, or using an Inter-Process Communication (IPC).
49 . The method according to claim 48 , wherein the IPC is using a file sharing, a signal, a socket, a pipe, a message queue, a shared memory, a semaphore, or memory mapped file, or wherein the IPC is using a clipboard, a Component Object Model (COM), a data copy, a DDE protocol, or mailslots.
50 . The method according to claim 1 , wherein the communication over the Internet between the requesting device and one or more of the selected devices is based on, uses, or is compatible with, Socket Secure (SOCKS) protocol or connection.
51 . The method according to claim 50 , wherein the Socks protocol or connection is according to, based on, or is compatible with, SOCKS4, SOCKS4a, or SOCKS5, or wherein the SOCKS protocol or connection is according to, based on, or is compatible with, IETF RFC 1928, IETF RFC 1929, IETF RFC 1961, or IETF RFC 3089.
52 . The method according to claim 1 , wherein the communication over the Internet between the requesting device and one or more of the selected devices is based on, uses, or is compatible with, WebSocket (ws) or WebSocket Secure (wss) protocol or connection.
53 . The method according to claim 1 , further comprising storing, operating, or using, by the requesting device, a web browser.
54 . The method according to claim 53 , wherein the web browser consists of, comprises of, or is based on, Microsoft Internet Explorer, Google Chrome, Opera™, or Mozilla Firefox®.
55 . The method according to claim 53 , wherein the web browser consists of, comprises of, or is based on, a mobile web browser.
56 . The method according to claim 55 , wherein the mobile web browser consists of, comprises of, or is based on, Safari, Opera Mini™, or Android web browser.