Systems and methods to control web scraping
View Patent ↗Systems and methods to control web scraping through a plurality of web servers using real time access statistics are described.
1. A method comprising:
providing a web interface to access the information stored in a first database, the web interface including a plurality of web servers capable to independently serve a current web request through retrieving the information from the first database;
in response to the current web request received at one of the web servers, determining identification information indicating a requester of the current web request;
logging the current web request in association with the requester in a second database, wherein the second database provides real time statistics on web requests received at the plurality of web servers;
accessing the second database, in response to the current web request, for past web requests of the requester logged in the second database, where a portion of the past web requests of the requester were received via each server of the plurality of web servers;
determining whether the requester is a web scraper based on whether a characteristic of the past web requests logged in the second database exceeds a threshold, wherein the characteristic of the past web requests logged in the second database is based at least in part on a pattern of unique data accesses over a period of time;
determining a type of processing based on the characteristic of the past web requests logged in the second database, such that the type of processing is selected from the list: block the current web request, delay the current web request, and timely reply to the current web request; wherein the characteristic of the past web requests logged in the second database is based at least in part on a pattern of unique data accesses over a period of time;
providing the information to the requester through one of the plurality of web servers where the type of processing is to timely reply to the current web request; and
delaying provision of the information to the requester through one of web servers where the type of processing is to delay the current web request.
2. The method of claim 1 , wherein the characteristic of the past web requests logged in the second database is based on a count of web requests within a predetermined period of time from the current web request.
3. The method of claim 1 , wherein the characteristic of the past web requests logged in the second database is based on an average speed of web requests generated by the requester.
4. The method of claim 1 , wherein each of the web servers logs received web requests in the second database; and each of the web servers accesses real time statistics of web requests received across the plurality of web servers to determine whether the requester is a web scraper.
5. The method of claim 4 , where the requester is a web scraper, the current web request is routed to separate web server that is not included in the plurality of web servers.
6. The method of claim 1 , further comprising:
determining a category of the current web request; and
selecting the threshold based on the category of the current web request.
7. The method of claim 1 , further comprising:
determining whether the requester is a known, friendly scraper based on the identification information;
when the requester is a friendly scraper, determining whether to slow the request based on current workload of the web interface.
8. The method of claim 7 , further comprising:
when the requester is a friendly scraper, determining whether to slow the request based on the characteristic of the past web requests.
9. The method of claim 1 , further comprising:
determining a category of the requester, wherein the category of the requester is selected from a list comprising: known scraper and/or end user; and
selecting a format to provide the information based on the category of the requester.
10. The method of claim 1 , wherein the accessing of the second database for the past web requests of the requester logged in the second database comprises:
counting web requests of the requester logged in the second database by the web servers within a predetermined period of time prior to the current web request.
11. The method of claim 1 , wherein the identification information indicating the requester of the current web request comprises an Internet address and an identification of user agent.
12. The method of claim 11 , further comprising:
extracting a user agent string from the current web request to determine the identification of the user agent.
13. The method of claim 1 , further comprising:
collecting an access pattern of average users; and
selecting the threshold based on the access pattern of average users.
14. A non-transitory tangible machine readable medium, storing instructions that, when executed by a computing device, cause the computing device to perform a method, the method comprising:
receiving a current web request at a web interface to access information stored on a first database, the web interface including a plurality of web servers capable to independently serve web requests through retrieving the information from the first database;
in response to the current web request received at one of the web servers, determining identification information indicating a requester of the current web request;
logging the current web request in association with the requester in a second database, wherein the second database provides real time statistics on web requests received at the plurality of web servers;
accessing the second database, in response to the current web request, for past web requests of the requester logged in the second database, where a portion of the past web requests of the requester were received via each server of the plurality of web servers;
determining whether the requester is a web scraper based on whether a characteristic of the past web requests logged in the second database exceeds a threshold, wherein the characteristic of the past web requests logged in the second database is based at least in part on a pattern of unique data accesses over a period of time;
determining a type of processing based on the characteristic of the past web requests logged in the second database, such that the type of processing is selected from the list: block the current web request, delay the current web request and timely reply to the current web request; wherein the characteristic of the past web requests logged in the second database is based at least in part on a pattern of unique data accesses over a period of time;
providing the information to the requestor through one of the plurality of web servers where the type of processing is to timely reply to the current web request; and
delaying provision of the information to the requester through one of web servers where the type of processing is to delay the current web request.
15. A data processing system, comprising:
means for receiving a current web request at a web interface to access information stored on a first database, the web interface including a plurality of web servers capable to independently serve web requests through retrieving the information from the first database;
means for determining identification information indicating a requester of the current web request, in response the current web request received at one of the web servers;
means for logging the current web request in association with the requester in a second database, in response to the current web request received at one of the web servers, wherein the second database provides real time statistics on web requests received at the plurality of web servers;
means for accessing the second database, in response to the current web request, for past web requests of the requester logged in the second database, where a portion of the past web requests of the requester were received via each server of the plurality web servers;
means for determining whether the requester is a web scraper based on whether a characteristic of the past web requests logged in the second database exceeds a threshold, wherein the characteristic of the past web requests logged in the second database is based at least in part on a pattern of unique data accesses over a period of time;
means for determining a type of processing based on the characteristic of the past web requests logged in the second database, such that the type of processing is selected from the list: block the current web request, delay the current web requested and timely reply to the current web request; wherein the characteristic of the past web requests logged in the second database is based at least in part on a pattern of unique data accesses over a period of time;
means for providing the information to the requester through one of the plurality of web servers, where the type of processing is to timely reply to the current web request; and
means for delaying provision of the information to the requester through one of web servers where the type of processing is to delay the current web request.
16. The method of claim 1 , where the type of processing is to delay the current web request, the current web request is routed to separate web server that is not included in the plurality of web servers.
17. The method of claim 1 , wherein the real time statistics reflect a pattern of data access for a plurality of requesters.
18. The method of claim 1 , wherein the real time statistics are used to prioritize responses to different requests.
19. The method of claim 1 . wherein the second database is distributed across respective random access memory in each of the plurality of web servers, such that different portions of past requests are stored on each of the plurality of web servers, wherein the plurality of web servers communicate with one another to obtain access to past requests.