System and method for near real time web scraping
A system and method perform near real time scraping of a web page to generate a modified web page. The method includes receiving a uniform resource locator (URL) request directed at a web server; detecting in the URL request a textual resource and a non-textual resource; generating a request for the textual resource over a first network path including a first determined latency; generating a request for the non-textual resource over a second network path including a second determined latency, which is greater than the first determined latency; receiving the textual resource and the non-textual resource; scraping data from the textual resource; and generating a modified web page based on the scraped data, the textual resource, and the non-textual resource.
1 . A method for near real time scraping of a web page to generate a modified web page, comprising:
receiving a uniform resource locator (URL) request directed at a web server;
detecting in the URL request a textual resource and a non-textual resource;
generating a request for the textual resource over a first network path including a first determined latency;
generating a request for the non-textual resource over a second network path including a second determined latency, which is greater than the first determined latency, wherein the second network path is different from the first network path;
receiving the textual resource;
scraping data from the textual resource prior to receiving the non-textual resource; and
generating a modified web page based on the scraped data, the textual resource, and the non-textual resource.
2 . The method of claim 1 , wherein the second network path includes a proxy server, the proxy server associated with a geolocation which is not a geolocation of a client device from which the URL request originates.
3 . The method of claim 1 , wherein scraping is performed in response to detecting a document object model (DOM) event.
4 . The method of claim 3 , wherein the DOM event indicates that the textual resource is received.
5 . The method of claim 1 , wherein the non-textual resource is any one of: an image, a video, a stylesheet, a media file, and any combination thereof.
6 . The method of claim 1 , further comprising:
detecting a predefined data field in the textual resource; and
detecting a value corresponding to the predefined data field.
7 . The method of claim 6 , further comprising:
generating the modified web page further based on the detected value.
8 . The method of claim 6 , further comprising:
querying an external database based on the detected value.
9 . The method of claim 8 , further comprising:
generating an additional content for the modified web page based on a result received in response to executing the query on the external database.
10 . The method of claim 1 , further comprising:
filtering a content from the non-textual resource, wherein the filtered content is not requested in the generated requests.
11 . The method of claim 10 , wherein the content is any one of: a JavaScript code, an image, a video, a multimedia, a font, an Ajax request, and a combination thereof.
12 . The method of claim 1 , wherein the first network path is based on a first geolocation, and the second network path is based on a second geolocation.
13 . A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to execute a process, the process comprising:
receiving a uniform resource locator (URL) request directed at a web server;
detecting in the URL request a textual resource and a non-textual resource;
generating a request for the textual resource over a first network path including a first determined latency;
generating a request for the non-textual resource over a second network path including a second determined latency, which is greater than the first determined latency, wherein the second network path is different from the first network path;
receiving the textual resource;
scraping data from the textual resource prior to receiving the non-textual resource; and
generating a modified web page based on the scraped data, the textual resource, and the non-textual resource.
14 . A system for near real time scraping of a web page to generate a modified web page, comprising:
a processing circuitry; and
a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to:
receive a uniform resource locator (URL) request directed at a web server;
detect in the URL request a textual resource and a non-textual resource;
generate a request for the textual resource over a first network path including a first determined latency;
generate a request for the non-textual resource over a second network path including a second determined latency, which is greater than the first determined latency, wherein the second network path is different from the first network path;
receive the textual resource;
scrape data from the textual resource prior to receiving the non-textual resource; and
generate a modified web page based on the scraped data, the textual resource, and the non-textual resource.
15 . The system of claim 14 , wherein the second network path includes a proxy server, the proxy server associated with a geolocation which is not a geolocation of a client device from which the URL request originates.
16 . The system of claim 14 , wherein the memory contain further instructions which when executed by the processing circuitry further configure the system to:
scrape data in response to detecting a document object model (DOM) event.
17 . The system of claim 16 , wherein the DOM event indicates that the textual resource is received.
18 . The system of claim 14 , wherein the non-textual resource is any one of:
an image, a video, a stylesheet, a media file, and any combination thereof.
19 . The system of claim 14 , wherein the memory contain further instructions which when executed by the processing circuitry further configure the system to:
detect a predefined data field in the textual resource; and
detect a value corresponding to the predefined data field.
20 . The system of claim 19 , wherein the memory contain further instructions which when executed by the processing circuitry further configure the system to:
generate the modified web page further based on the detected value.
21 . The system of claim 19 , wherein the memory contain further instructions which when executed by the processing circuitry further configure the system to:
query an external database based on the detected value.
22 . The system of claim 21 , wherein the memory contain further instructions which when executed by the processing circuitry further configure the system to:
generate an additional content for the modified web page based on a result received in response to executing the query on the external database.
23 . The system of claim 14 , wherein the memory contain further instructions which when executed by the processing circuitry further configure the system to:
filter a content from the non-textual resource, wherein the filtered content is not requested in the generated requests.
24 . The system of claim 23 , wherein the content is any one of: a JavaScript code, an image, a video, a multimedia, a font, an Ajax request, and a combination thereof.
25 . The system of claim 14 , wherein the first network path is based on a first geolocation, and the second network path is based on a second geolocation.