System and method for near real time web scraping

1 . A method for near real time scraping of a web page to generate a modified web page, comprising:

receiving a uniform resource locator (URL) request directed at a web server;

detecting in the URL request a textual resource and a non-textual resource;

generating a request for the textual resource over a first network path including a first determined latency;

generating a request for the non-textual resource over a second network path including a second determined latency, which is greater than the first determined latency, wherein the second network path is different from the first network path;

receiving the textual resource;

scraping data from the textual resource prior to receiving the non-textual resource; and

generating a modified web page based on the scraped data, the textual resource, and the non-textual resource.

2 . The method of claim 1 , wherein the second network path includes a proxy server, the proxy server associated with a geolocation which is not a geolocation of a client device from which the URL request originates.

3 . The method of claim 1 , wherein scraping is performed in response to detecting a document object model (DOM) event.

4 . The method of claim 3 , wherein the DOM event indicates that the textual resource is received.

5 . The method of claim 1 , wherein the non-textual resource is any one of: an image, a video, a stylesheet, a media file, and any combination thereof.

6 . The method of claim 1 , further comprising:

detecting a predefined data field in the textual resource; and

detecting a value corresponding to the predefined data field.

7 . The method of claim 6 , further comprising:

generating the modified web page further based on the detected value.

8 . The method of claim 6 , further comprising:

querying an external database based on the detected value.

9 . The method of claim 8 , further comprising:

generating an additional content for the modified web page based on a result received in response to executing the query on the external database.

10 . The method of claim 1 , further comprising:

filtering a content from the non-textual resource, wherein the filtered content is not requested in the generated requests.

11 . The method of claim 10 , wherein the content is any one of: a JavaScript code, an image, a video, a multimedia, a font, an Ajax request, and a combination thereof.

12 . The method of claim 1 , wherein the first network path is based on a first geolocation, and the second network path is based on a second geolocation.

13 . A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to execute a process, the process comprising:

receiving a uniform resource locator (URL) request directed at a web server;

detecting in the URL request a textual resource and a non-textual resource;

generating a request for the textual resource over a first network path including a first determined latency;

generating a request for the non-textual resource over a second network path including a second determined latency, which is greater than the first determined latency, wherein the second network path is different from the first network path;

receiving the textual resource;

scraping data from the textual resource prior to receiving the non-textual resource; and

generating a modified web page based on the scraped data, the textual resource, and the non-textual resource.

14 . A system for near real time scraping of a web page to generate a modified web page, comprising:

a processing circuitry; and

a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to:

receive a uniform resource locator (URL) request directed at a web server;

detect in the URL request a textual resource and a non-textual resource;

generate a request for the textual resource over a first network path including a first determined latency;

generate a request for the non-textual resource over a second network path including a second determined latency, which is greater than the first determined latency, wherein the second network path is different from the first network path;

receive the textual resource;

scrape data from the textual resource prior to receiving the non-textual resource; and

generate a modified web page based on the scraped data, the textual resource, and the non-textual resource.

15 . The system of claim 14 , wherein the second network path includes a proxy server, the proxy server associated with a geolocation which is not a geolocation of a client device from which the URL request originates.

16 . The system of claim 14 , wherein the memory contain further instructions which when executed by the processing circuitry further configure the system to:

scrape data in response to detecting a document object model (DOM) event.

17 . The system of claim 16 , wherein the DOM event indicates that the textual resource is received.

18 . The system of claim 14 , wherein the non-textual resource is any one of:

an image, a video, a stylesheet, a media file, and any combination thereof.

19 . The system of claim 14 , wherein the memory contain further instructions which when executed by the processing circuitry further configure the system to:

detect a predefined data field in the textual resource; and

detect a value corresponding to the predefined data field.

20 . The system of claim 19 , wherein the memory contain further instructions which when executed by the processing circuitry further configure the system to:

generate the modified web page further based on the detected value.

21 . The system of claim 19 , wherein the memory contain further instructions which when executed by the processing circuitry further configure the system to:

query an external database based on the detected value.

22 . The system of claim 21 , wherein the memory contain further instructions which when executed by the processing circuitry further configure the system to:

generate an additional content for the modified web page based on a result received in response to executing the query on the external database.

23 . The system of claim 14 , wherein the memory contain further instructions which when executed by the processing circuitry further configure the system to:

filter a content from the non-textual resource, wherein the filtered content is not requested in the generated requests.

24 . The system of claim 23 , wherein the content is any one of: a JavaScript code, an image, a video, a multimedia, a font, an Ajax request, and a combination thereof.

25 . The system of claim 14 , wherein the first network path is based on a first geolocation, and the second network path is based on a second geolocation.