Systems and methods to control web scraping
View Patent ↗Systems and methods to control web scraping through a plurality of web servers using real time access statistics are described. For example, in one embodiment a web request is categorized based at least in part on a type of data access. The web request can be processed based on threshold determinations. The processing can include blocking, delaying, timely replying, or prioritizing a response to the web request.
1. A system comprising:
one or more network interfaces accessible from a network;
one or more repositories to retain one or more of:
web content;
requester identification information; and/or
web request history information;
one or more processors coupled to at least one of the one or more network interfaces and to at least one of the one or more repositories, the one or more processors to execute instructions to:
process a current web request that is received via the network and corresponds to a request for information;
determine a request category corresponding to the current web request based at least in part on a type of data access corresponding to the current web request;
determine a threshold based on the request category of the current web request, wherein the threshold is from a plurality of thresholds, each of which thresholds corresponds to a respective request category of web requests;
determine identification information indicating a requester of the current web request;
log the current web request in association with the requester;
process information on past web requests of the requester to identify a characteristic of the past web requests;
determine whether the requester of the current web request is a web scraper based on whether the characteristic of the past web requests exceeds the threshold;
provide types of processing to handle the current web request wherein the provided types of processing include at least blocking, delaying, timely replying, and prioritizing a response to the current web request:
determine a type of processing based on the determining whether the requester of the current web request is a web scraper; and
handle the current web request based on the type of processing wherein (a) blocking comprises not responding to the current web request, (b) delaying comprises postponing a response to the current web request for a predetermined period of time, (c) timely replying comprises providing a timely response to the current web request, and (d) prioritizing comprises indicating a priority for a response to the current web request.
2. The system of claim 1 , wherein the determining the request category corresponding to the current web request comprises:
characterizing the current web request as corresponding to a type of web request based on a type of information requested.
3. The system of claim 1 , wherein the threshold is based at least in part on access pattern information for web requests corresponding to the category of the current web request.
4. The system of claim 3 , wherein the web requests correspond to a type of web page requested.
5. The system of claim 1 , wherein the prioritizing further comprises:
characterizing the current web request as corresponding to a type of web request based on a type of information requested.
6. The system of claim 1 , wherein the one or more processors are to further execute instructions to:
determine whether the requester is a friendly scraper.
7. The system of claim 6 , wherein the one or more processors are to further execute instructions to:
consequent to a determination that the requester is a friendly scraper, determine the type of processing based on a priority of a second requester that is not determined to be a web scraper.
8. The system of claim 7 , wherein the second requester that is not determined to be a web scraper is determined to correspond to a newly identified requester.
9. The system of claim 7 , wherein the second requester that is not determined to be a web scraper is determined to have a low request rate and/or a low number of requests.
10. A method comprising:
processing a current web request that is received via the Internet and corresponds to a request for information retained in one or more repositories;
determining a request category corresponding to the current web request based at least in part on a type of data access corresponding to the current web request;
determining a threshold based on the request category of the current web request, wherein the threshold is from a plurality of thresholds, each of which thresholds corresponds to a respective request category of web requests;
determining identification information indicating a requester of the current web request;
logging the current web request in association with the requester;
processing information on past web requests of the requester to identify a characteristic of the past web requests;
determining whether the requester of the current web request is a web scraper based on whether the characteristic of the past web requests exceeds the threshold;
providing types of processing to handle the current web request wherein the provided types of processing include at least blocking, delaying, timely replying, and prioritizing a response to the current web request;
determining a type of processing based on the determining whether the requester of the current web request is a web scraper; and
handling the current web request based on the type of processing wherein (a) blocking comprises not responding to the current web request, (b) delaying comprises postponing a response to the current web request for a predetermined period of time, (c) timely replying comprises providing a timely response to the current web request, and (d) prioritizing comprises indicating a priority for a response to the current web request.
11. The method of claim 10 , wherein the determining the request category corresponding to the current web request comprises:
characterizing the current web request as corresponding to a type of web request based on a type of information requested.
12. The method of claim 10 , wherein the threshold is based at least in part on access pattern information for web requests corresponding to the category of the current web request.
13. The method of claim 12 , wherein the web requests correspond to a type of web page requested.
14. The method of claim 10 , wherein the prioritizing further comprises:
characterizing the current web request as corresponding to a type of web request based on a type of information requested.
15. The method of claim 10 , further comprising:
determining whether the requester is a friendly scraper;
consequent to a determination that the requester is a friendly scraper, determining the type of processing based on a priority of a second requester that is not determined to be a web scraper.
16. The method of claim 15 , wherein the second requester that is not determined to be a web scraper is determined to correspond to a newly identified requester, to a low request rate, and/or to a low number of requests.
17. A non-transitory machine-readable medium having machine-readable instructions thereon, which instructions, when executed by one or more computers or other processing devices, cause the one or more one or more computers or other processing devices to:
process a current web request that is received via the network and corresponds to a request for information;
determine a request category corresponding to the current web request based at least in part on a type of data access corresponding to the current web request;
determine a threshold based on the request category of the current web request, wherein the threshold is from a plurality of thresholds, each of which thresholds corresponds to a respective request category of web requests;
determine identification information indicating a requester of the current web request;
log the current web request in association with the requester;
process information on past web requests of the requester to identify a characteristic of the past web requests;
determine whether the requester of the current web request is a web scraper based on whether the characteristic of the past web requests exceeds the threshold;
provide types of processing to handle the current web request wherein the provided types of processing include at least blocking, delaying, timely replying, and prioritizing a response to the current web request:
determine a type of processing based on the determining whether the requester of the current web request is a web scraper; and
handle the current web request based on the type of processing wherein (a) blocking comprises not responding to the current web request, (b) delaying comprises postponing a response to the current web request for a predetermined period of time, (c) timely replying comprises providing a timely response to the current web request, and (d) prioritizing comprises indicating a priority for a response to the current web request.
18. The non-transitory machine-readable medium of claim 17 , wherein the determining the request category corresponding to the current web request comprises:
characterizing the current web request as corresponding to a type of web request based on a type of information requested.
19. The non-transitory machine-readable medium of claim 17 , wherein the threshold is based at least in part on access pattern information for web requests corresponding to the category of the current web request.
20. The non-transitory machine-readable medium of claim 19 , wherein the web requests correspond to a type of web page requested.