Method and apparatus to throttle media access by web crawlers
View Patent ↗Methods, apparatus, systems and articles of manufacture are disclosed to throttle resource access by web crawlers. An example apparatus disclosed herein includes a data interface to obtain a media request message for media hosted by a server, the media request message requesting access to the media. The example apparatus also includes a category handler to determine a source indicator of a media-requesting source associated with the media request message, the source indicator indicating whether the media-requesting source is associated with a media provider to provide audience measurement information associated with the media provided by the media provider. The example apparatus also includes a penalty manager to insert a time delay including a delay tag in the media response message to the media-requesting source based on the source indicator, the delay tag to cause a delay time period based on the source indicator, at least one of the data interface, category handler, and penalty manager is implemented on a logic circuit.
1. An apparatus comprising:
a data interface to obtain a media request message for media hosted by a server, the media request message requesting access to the media;
a category handler to determine a source indicator of a media-requesting source associated with the media request message, the source indicator indicating whether the media-requesting source is associated with a media provider to provide audience measurement information associated with the media provided by the media provider; and
a penalty manager to insert a time delay including a delay tag in the media response message to the media-requesting source based on the source indicator, the delay tag to cause a delay time period based on the source indicator, at least one of the data interface, category handler, and penalty manager is implemented on a logic circuit.
2. The apparatus as defined in claim 1 , wherein the media-requesting source is a web crawler.
3. The apparatus as defined in claim 1 , wherein the delay tag includes executable instructions.
4. The apparatus as defined in claim 3 , wherein the executable instructions include a native delay function.
5. The apparatus as defined in claim 1 , wherein execution of the delay tag generates the time delay.
6. The apparatus as defined in claim 1 , wherein execution of the delay tag requests data from a web site associated with generating the time delay.
7. The apparatus as defined in claim 1 , wherein the penalty manager is to insert the time delay in the media response message by selecting a pre-existing copy of the media with a delay tag associated with the time delay.
8. The apparatus as defined in claim 1 , wherein the media request message is either a hypertext transfer protocol GET request or a hypertext transfer protocol POST request.
9. An apparatus to throttle media access at a server, the method comprising:
means for obtaining, at the server, a media request message for media hosted by the server, the media request message requesting access to the media;
means for determining a source indicator of a media-requesting source, the source indicator indicating whether the media-requesting source is associated with a media provider to provide audience measurement information associated with the media provided by the media provider; and
means for inserting a time delay including a delay tag in the media response message to the media-requesting source based on the source indicator, the delay tag to cause a delay time period.
10. The apparatus as defined in claim 9 , wherein the media-requesting source is a web crawler.
11. The apparatus as defined in claim 9 , wherein the delay tag includes executable instructions.
12. The apparatus as defined in claim 11 , wherein the executable instructions include a native delay function.
13. The apparatus as defined in claim 9 , wherein executing the delay tag generates the time delay.
14. The apparatus as defined in claim 9 , wherein executing the delay tag causes the media-requesting source to request data from a web site associated with generating the time delay.
15. The apparatus as defined in claim 9 , wherein inserting the time delay in the media response message comprises selecting a pre-existing copy of the media with a delay tag associated with the time delay.
16. The apparatus as defined in claim 9 , wherein the media request message is either a hypertext transfer protocol GET request or a hypertext transfer protocol POST request.
17. A non-transitory tangible computer readable storage medium comprising instructions that, when executed, cause a server hosting media to at least:
obtain a media request message for media hosted by the server, the media request message requesting access to the media;
determine a source indicator of a media-requesting source, the source indicator indicating whether the media-requesting source is associated with a media provider to provide audience measurement information associated with the media provided by the media provider; and
insert a time delay including a delay tag in the media response message to the media-requesting source based on the characterization, the delay tag to cause a delay time period based on the source indicator.
18. The computer readable storage medium as defined in claim 17 , wherein executing the delay tag causes the media-requesting source to request data from a web site associated with generating the time delay.
19. The computer readable storage medium as defined in claim 17 , wherein the instructions further cause the server to insert the time delay in the media response message by selecting a pre-existing copy of the media with a delay tag associated with the time delay.
20. The computer readable storage medium as defined in claim 17 , wherein the media request message is either a hypertext transfer protocol GET request or a hypertext transfer protocol POST request.