Method and apparatus to throttle media access by web crawlers
View Patent ↗Methods, apparatus, systems and articles of manufacture are disclosed to throttle resource access by web crawlers. An example method disclosed herein includes obtaining, at a server, a media request message for media hosted by the server, the media request message requesting access to the media, characterizing a media-requesting source associated with the media request message, and inserting a time delay in a media response message to the media-requesting source based on the characterization.
1. A method to throttle media access at a server, the method comprising:
obtaining, at the server, a media request message for media hosted by the server, the media request message requesting access to the media;
extracting source-identifying information from the media request message to identify a media-requesting source associated with the media request message;
matching the source-identifying information with a source category identifier from a data collection facility in a lookup table with the source-identifying information to determine a partner affiliation status of the media-requesting source, the partner affiliation status indicating whether the media-requesting source has a business partnership with a media provider to provide audience measurement information associated with the server;
and
inserting a time delay including a delay tag in the media response message to the media-requesting source based on the partner affiliation status, the delay tag to cause a delay time period.
2. The method as defined in claim 1 , wherein the media-requesting source is a web crawler.
3. The method as defined in claim 1 , wherein the delay tag includes executable instructions.
4. The method as defined in claim 3 , wherein the executable instructions include a native delay function.
5. The method as defined in claim 1 , wherein executing the delay tag generates the time delay.
6. The method as defined in claim 1 , wherein executing the delay tag causes the media-requesting source to request data from a web site associated with generating the time delay.
7. The method as defined in claim 1 , wherein inserting the time delay in the media response message comprises selecting a pre-existing copy of the media with a delay tag associated with the time delay.
8. The method as defined in claim 1 , wherein the media request message is either a hypertext transfer protocol GET request or a hypertext transfer protocol POST request.
9. A system comprising:
a data interface to obtain a media request message for media hosted by a server, the media request message requesting access to the media;
a category handler to determine a media-requesting source associated with the media request message and a partner affiliation status of the media-requesting source, the partner affiliation status indicating whether the media-requesting source has a business partnership with a media provider to provide audience measurement information associated with the server, and extract source-identifying information from the media request message, match the source-identifying information with a source category identifier from a data collection facility in a lookup table with the source-identifying information;
a penalty manager to insert a time delay including a delay tag in the media response message to the media-requesting source based on the partner affiliation status, the delay tag to cause a delay time period based on the source-identifying information at least one of the data interface, category handler, and penalty manager is implemented on a logic circuit.
10. The system as defined in claim 9 , wherein the media-requesting source is a web crawler.
11. The system as defined in claim 9 , wherein the delay tag includes executable instructions.
12. The system as defined in claim 11 , wherein the executable instructions include a native delay function.
13. The system as defined in claim 9 , wherein execution of the delay tag generates the time delay.
14. The system as defined in claim 9 , wherein execution of the delay tag requests data from a web site associated with generating the time delay.
15. The system as defined in claim 9 , wherein the penalty manager is to insert the time delay in the media response message by selecting a pre-existing copy of the media with a delay tag associated with the time delay.
16. The system as defined in claim 9 , wherein the media request message is either a hypertext transfer protocol GET request or a hypertext transfer protocol POST request.
17. A tangible computer readable storage medium comprising instructions that, when executed, cause a server hosting media to at least:
obtain a media request message for media hosted by the server, the media request message requesting access to the media;
extract source-identifying information from the media request message;
match the source-identifying information with a source category identifier from a data collection facility in a lookup table with the source-identifying information to determine a partner affiliation status of the media-requesting source, the partner affiliation status indicating whether the media-requesting source has a business partnership with a media provider to provide audience measurement information associated with the server; and
insert a time delay including a delay tag in the media response message to the media-requesting source based on the characterization, the delay tag to cause a delay time period based on the source-identifying information.
18. The tangible computer readable storage medium as defined in claim 17 , wherein the instructions further cause the server to insert the time delay in the media response message by selecting a pre-existing copy of the media with a delay tag associated with the time delay.
19. A method to throttle media access at a server, the method comprising:
obtaining, at the server, a media request message for media hosted by the server, the media request message requesting access to the media;
extracting source-identifying information from the media request message to identify a media-requesting source associated with the media request message;
matching the source-identifying information with a source category identifier from a data collection facility in a lookup table with the source-identifying information to determine a partner affiliation status of the media-requesting source;
determining a time delay for a media response message based on the partner affiliation status; and
inserting the time delay including a delay tag in the media response message to the media-requesting source based on the partner affiliation status, the delay tag to cause a delay time period,
wherein the partner affiliation status indicates whether the media-requesting source has a business partnership with a media provider to provide audience measurement information associated with the server.