Methods and apparatus to throttle media access by web crawlers
Methods, apparatus, systems and articles of manufacture are disclosed to throttle resource access by web crawlers. An example method disclosed herein includes obtaining, at a server, a media request message for media hosted by the server, the media request message requesting access to the media, characterizing a media-requesting source associated with the media request message, and inserting a time delay in a media response message to the media-requesting source based on the characterization.
1. An apparatus comprising:
message logger circuitry to:
identify source-identifying information corresponding to a sender of a media request message, the media request message corresponding to media hosted by a server, the media request message including a request for access to the media; and
associate the media request message with a label indicating capabilities of the sender; and
randomizer circuitry to make a determination of a choice selected from (a) whether to enforce a server-side delay for access to the media and (b) whether to enforce a sender-side delay for access to the media, the determination based on the label associated with the media request message.
2. The apparatus as defined in claim 1 , wherein the message logger circuitry is to identify whether the sender can execute executable instructions associated with the media.
3. The apparatus as defined in claim 2 , further including delay enforcer circuitry to enforce the server-side delay for access to the media when the sender is unable to execute the executable instructions associated with the media.
4. The apparatus as defined in claim 3 , wherein the delay enforcer circuitry is to cause a delay time period prior to sending the media to the sender of the media request message.
5. The apparatus as defined in claim 2 , further including tag embedder circuitry to embed the executable instructions associated with the media, the executable instructions including a delay time period to occur prior to presenting the media.
6. The apparatus as defined in claim 5 , wherein the tag embedder circuitry is to adjust the delay time period based on characteristics corresponding to the sender of the media request message.
7. An apparatus to throttle media access at a media source device, the apparatus comprising:
means for obtaining a media request message corresponding to media hosted by the media source device, the media request message including a request for access to the media;
means for logging to:
identify source-identifying information corresponding to a sender of the media request message; and
associate the media request message with a label indicating capabilities of the sender; and
means for making a determination of a choice selected from (a) whether to enforce a media source-side delay for access to the media and (b) whether to enforce a sender-side delay for access to the media, the determination based on the label associated with the media request message.
8. The apparatus as defined in claim 7 , wherein the logging means is to identify whether the sender can execute executable instructions associated with the media.
9. The apparatus as defined in claim 8 , further including means for delay enforcement to enforce a source-side delay for access to the media when the sender is unable to execute the executable instructions associated with the media.
10. The apparatus as defined in claim 9 , wherein the delay enforcement means is to cause a delay time period by a server prior to sending the media to the sender of the request.
11. The apparatus as defined in claim 8 , further including means for embedding to embed a tag containing the executable instructions associated with the media, the tag including a delay time period to occur by the sender of the media request message prior to presenting the media.
12. The apparatus as defined in claim 11 , wherein the embedding means is to adjust the delay time period based on characteristics corresponding to the sender of the media request message.
13. A non-transitory computer readable storage medium comprising instructions that, when executed, cause a processor to at least:
identify source-identifying information corresponding to a sender of a media request message, the media request message corresponding to media hosted by a server, the media request message including a request for access to the media;
associate the media request message with a label indicating capabilities of the sender; and
make a determination of a choice selected from (a) whether to enforce a server-side delay for access to the media and (b) whether to enforce a sender-side delay for access to the media, the determination based on the label associated with the media request message.
14. The computer readable storage medium as defined in claim 13 , wherein the instructions, when executed, cause the processor to identify whether the sender can execute executable instructions associated with the media.
15. The computer readable storage medium as defined in claim 14 , wherein the instructions, when executed, cause the processor to enforce the server-side delay for access to the media when the sender is unable to execute the executable instructions associated with the media.
16. The computer readable storage medium as defined in claim 13 , wherein the instructions, when executed, cause the processor to cause a delay time period prior to sending the media to the sender of the media request message.
17. The computer readable storage medium as defined in claim 14 , wherein the instructions, when executed, cause the processor to embed the executable instructions associated with the media, the tag executable instructions including a delay time period to occur prior to presenting the media.
18. The computer readable storage medium as defined in claim 17 , wherein the instructions, when executed, cause the processor to adjust the delay time period based on characteristics corresponding to the sender of the media request message.