System and method for managing non-direct URL fetching service
A method for providing and managing non-direct URL fetching service for retrieving a content from a web server to a client device is disclosed, such as for overcoming geo-blocking or a Man-In-The-Middle (MITM) attack. The non-direct fetching method may use intermediate devices, such as proxy server, Data-Center proxy server, tunnel devices, or any combination thereof. A URL request may be sent in parallel using both direct and non-direct fetching schemes, in order to verify the need for using the non-direct fetching service. Direct or non-direct fetching scheme may be selected by using a file that associates a fetching scheme to the requested URL. The selection of the fetching mechanism may use dynamically in real-time updating of a Proxy Auto-Configuration (PAC) file. As part of an accounting scheme, quotas may be applied to a cumulative received data or a time duration of using a non-direct fetching service.
1 . A method for obtaining, by a first device, a first content identified by a first Uniform Resource Locator (URL) and stored in a first web server, and a second content identified by a second Uniform Resource Locator (URL) and stored in a second web server, for use with a first database that comprises multiple countries and that associates a respective value to each of the countries, for use with a second database that comprises multiple countries and that associates a respective value to each of the countries, and for use with a group of devices, each of the devices in the group is associated with a respective country, the method comprising:
identifying, by the first device, the first URL;
selecting, by the first device, in response to the identifying of the first URL, a first country from the first database, based on the respective value associated with the first country;
increasing by 1 a value that is associated with the first country in the first database;
selecting, by the first device from the group, a second device that is associated with the first country;
sending, by the first device to the first web server via the selected second device over the Internet, a first Hypertext Transfer Protocol (HTTP) request that comprises the first URL, so that the first HTTP request that is received by the first web server does not include an Internet Protocol (IP) address of the first device as a source address;
receiving, by the first device from the first web server via the selected second device over the Internet, the first content, in response to the sending of the first HTTP request;
identifying, by the first device, the second URL;
selecting, by the first device, in response to the identifying of the second URL, a second country from the second database, based on the respective value associated with the second country;
increasing by 1 the value associated with the second country in the second database;
selecting, by the first device from the group, a third device that is associated with the second country;
sending, by the first device to the second web server via the selected third device over the Internet, a second Hypertext Transfer Protocol (HTTP) request that comprises the second URL, so that the second HTTP request that is received by the second web server does not include an Internet Protocol (IP) address of the first device as a source address; and
receiving, by the first device from the second web server via the selected third device over the Internet, the second content, in response to the sending of the second HTTP request.
2 . The method according to claim 1 , further comprising executing, by the first device, a web browser, wherein the identifying, of the first or second URL is performed as part of the executing of the web browser.
3 . The method according to claim 2 , further comprising storing, by the first device, the web browser.
4 . The method according to claim 2 , wherein the web browser comprises Microsoft Internet Explorer, Google Chrome, Opera™, or Mozilla Firefox®.
5 . The method according to claim 2 , wherein the web browser comprises a mobile web browser.
6 . The method according to claim 5 , wherein the mobile web browser comprises of, or is based on, Safari, Opera Mini™, or Android web browser.
7 . The method according to claim 1 , wherein an amount of distinct countries in the first or second database is equal to, or more than, 1, 2, 3, 4, 5, 7, 10, 12, 15, or 20.
8 . The method according to claim 1 , wherein the first or second database is stored in the first device.
9 . The method according to claim 1 , wherein the first or second database is stored in a server device, the method further comprising receiving, by the first device, the respective first or second database from the server.
10 . The method according to claim 1 , wherein the first device comprises a server device.
11 . The method according to claim 10 , wherein the server device is part of, or is integrated with, a proxy server.
12 . The method according to claim 11 , wherein the proxy server comprises an HTTP proxy server, a web-proxy server, a caching proxy, an open-source caching proxy server, a cloud-based proxy server, an open proxy server, a forwarding proxy server, a reverse proxy server, a transparent proxy server, a non-transparent proxy server, an anonymous proxy server, a translation proxy server, a SOCKS proxy server, a CGI web proxy server, a suffix proxy server, an I2P anonymous proxy server, a DNS proxy server, or any combination thereof.
13 . The method according to claim 1 , wherein the selecting of the first or second country is based on, or comprises, selecting by a user.
14 . The method according to claim 13 , wherein the selecting of the first country comprises:
displaying, to the user, the multiple countries and associated values for selecting therefrom; and
obtaining, from the user, the selected first country.
15 . The method according to claim 14 , wherein the displaying comprises displaying the countries having largest associated values.
16 . The method according to claim 1 , further comprising estimating a third country of the web server in response to the identifying of the first URL.
17 . The method according to claim 16 , wherein the estimating of the third country is based on, or uses, geolocation.
18 . The method according to claim 17 , wherein the geolocation is based on Internet Protocol (IP) geolocation.
19 . The method according to claim 18 , wherein the geolocation is based on W3C Geolocation Application Programming Interface (API).
20 . The method according to claim 16 , wherein the first country is selected based on the third country.
21 . The method according to claim 20 , wherein the third country is selected as the first country.
22 . The method according to claim 1 , wherein the selecting of the first or second country is based on, or uses, load balancing.
23 . The method according to claim 1 , wherein the selecting of the first or second country is based on, or uses, random selection.
24 . The method according to claim 23 , wherein random selection uses, or is based on, one or more random numbers generated by a random number generator.
25 . The method according to claim 24 , wherein the random number generator is hardware based.
26 . The method according to claim 25 , wherein the random number generator is using thermal noise, shot noise, nuclear decaying radiation, photoelectric effect, or quantum phenomena.
27 . The method according to claim 24 , wherein the random number generator is software based.
28 . The method according to claim 27 , wherein the random number generator is based on executing an algorithm for generating pseudo-random numbers.
29 . The method according to claim 1 , wherein the selecting of the first or second country is based on, or uses, the estimated country of the first device.
30 . The method according to claim 1 , further comprising estimating the country of each one of the devices in the group using geolocation.
31 . The method according to claim 30 , wherein the geolocation is based on TP geolocation.
32 . The method according to claim 26 , wherein the geolocation is based on W3C Geolocation Application Programming Interface (API).
33 . The method according to claim 26 , wherein the second or third device is selected based on being in the same area as the first device or as the first web server.
34 . The method according to claim 1 , wherein the second device is selected based on estimated as being in the country or city as the first device or the first web server.
35 . The method according to claim 1 , wherein the second device is further selected based on being a recent one to be selected, or based on being the least recent to be selected.
36 . The method according to claim 1 , wherein the second or third device is manually selected by a user.
37 . The method according to claim 36 , further comprising displaying to the user identifiers of the devices in the group, and selecting, by the user, the respective second or third device from the group.
38 . The method according to claim 1 , wherein the second or third device is a client device, and the method further comprising storing, operating, or using, by the first device, a client operating system.
39 . The method according to claim 1 , wherein the first device is a client device, and the method further comprising storing, operating, or using, by the first device, a client operating system.
40 . The method according to claim 39 , wherein the client operating system comprises one out of Microsoft Windows 7, Microsoft Windows XP, Microsoft Windows 8, Microsoft Windows 8.1, Linux, Google Chrome OS, or any combination thereof.
41 . The method according to claim 39 , wherein the client operating system is a Real-Time Operating System (RTOS).
42 . The method according to claim 40 , wherein the RTOS comprises FreeRTOS, SafeRTOS, QNX, VxWorks, or Micro-Controller Operating Systems (μC/OS).
43 . The method according to claim 1 , wherein the second or third device, comprises, or is part of, a wearable device that is wearable on a person.
44 . The method according to claim 1 , wherein the first device, comprises, or is part of, a wearable device that is wearable on a person.
45 . The method according to claim 44 , wherein the wearable device is wearable on an organ of the person head.
46 . The method according to claim 45 , wherein the organ is an eye, ear, face, cheek, nose, mouth, lip, forehead, or chin.
47 . The method according to claim 44 , wherein the wearable device is constructed to have a form substantially similar to, is constructed to have a shape allowing mounting or wearing identical or similar to, or is constructed to have a form to at least in part substitute for, headwear, eyewear, or earpiece.
48 . The method according to claim 44 , wherein the wearable device is shaped for permanently or releasably being attachable to, or be part of, a clothing piece of a person.
49 . The method according to claim 48 , wherein the attaching uses taping, gluing, pinning, enclosing, encapsulating, a pin, or a latch and hook clip.
50 . The method according to claim 49 , wherein the clothing piece is a top, bottom, or full-body underwear, or a headwear, a footwear, an accessory, an outwear, a suit, a dress, a skirt, or a top.
51 . The method according to claim 44 , wherein the wearable device further comprises an annular member defining an aperture therethrough that is sized for receipt therein of a part of a human body.
52 . The method according to claim 1 , further comprising executing, by the first device, a web browser, wherein the identifying, of the first or second URL is performed as part of the executing of the web browser, and wherein at least one of the steps is performed as integrated with the web browser in a form of a plug-in or an extension.
53 . The method according to claim 52 , wherein at least one of the steps performed by the first device is integrated with the web browser in a form of a plug-in or an extension.
54 . The method according to claim 52 , wherein the identifying of the URL uses a plug-in or an extension to the web browser.
55 . The method according to claim 52 , wherein the integration is by hooking to the web browser, wherein the integration is in a filter driver form, or, wherein the web browser and the steps are communicating using an Inter-Process Communication (IPC).
56 . The method according to claim 55 , wherein the identifying of the URL request uses hooking to the web browser, a filter driver form, or using an Inter-Process Communication (IPC).
57 . The method according to claim 55 , wherein the IPC is using a database sharing, a signal, a socket, a pipe, a message queue, a shared memory, a semaphore, or memory mapped file.
58 . The method according to claim 55 , wherein the IPC is using a clipboard, a Component Object Model (COM), a data copy, a DDE protocol, or mailslots.
59 . The method according to claim 1 , wherein the second or third device is integrated in part or entirely in an appliance.
60 . The method according to claim 1 , wherein the first device is integrated in part or entirely in an appliance.
61 . The method according to claim 60 , wherein a primary functionality of the appliance is associated with food storage, handling, or preparation.
62 . The method according to claim 61 , wherein a primary function of the appliance is heating food, and wherein the appliance is a microwave oven, an electric mixer, a stove, an oven, or an induction cooker.
63 . The method according to claim 61 , wherein the appliance is a refrigerator, a freezer, a food processor, a dishwasher, a food blender, a beverage maker, a coffeemaker, or an iced-tea maker.
64 . The method according to claim 60 , wherein a primary function of the appliance is associated with environmental control, and the appliance comprises or is part of, an HVAC system.
65 . The method according to claim 64 , wherein a primary function of the appliance is associated with temperature control, and wherein the appliance is an air conditioner or a heater.
66 . The method according to claim 60 , wherein a primary function of the appliance is associated with cleaning, wherein the primary function is associated with clothes cleaning, and the appliance is a washing machine or a clothes dryer, or wherein the appliance is a vacuum cleaner.
67 . The method according to claim 60 , wherein a primary function of the appliance is associated with water control or water heating.
68 . The method according to claim 60 , wherein the appliance is an answering machine, a telephone set, a home cinema method, a HiFi method, a CD or DVD player, an electric furnace, a trash compactor, a smoke detector, a light fixture, or a dehumidifier.
69 . The method according to claim 60 , wherein the appliance is a battery-operated portable electronic device, and the appliance is a notebook, a laptop computer, a media player, a cellular phone, a Personal Digital Assistant (PDA), an image processing device, a digital camera, a video recorder, or a handheld computing device.
70 . The method according to claim 1 , wherein the second or third device is housed in a single enclosure that is a hand-held enclosure or a portable enclosure.
71 . The method according to claim 1 , wherein the first device is housed in a single enclosure that is a hand-held enclosure or a portable enclosure.
72 . The method according to claim 71 , wherein the client device is further integrated with at least one of a notebook-computer, a laptop computer, a media player, a Digital Still Camera (DSC), a Digital video Camera (DVC or digital camcorder), a Personal Digital Assistant (PDA), a cellular telephone, a digital camera, a video recorder, or a smartphone.
73 . The method according to claim 72 , wherein the smartphone comprises, or is based on, an Apple iPhone 6 or a Samsung Galaxy S6.
74 . The method according to claim 1 , wherein the first or second content comprises a web-page or a part thereof.
75 . The method according to claim 74 , wherein the first content comprises a first web-page or a part thereof, and wherein the second content comprises a second web-page or a part thereof.
76 . A non-transitory computer readable medium containing computer instructions that, when executed by a computer processor, cause the processor to perform at least part of the steps of claim 1 .
77 . A non-transitory computer readable medium containing computer instructions that, when executed by a computer processor, cause the processor to perform all of the steps of claim 1 .