Methods and apparatus to throttle media access by web crawlers
Methods, apparatus, systems and articles of manufacture are disclosed to throttle resource access by web crawlers. An example apparatus disclosed herein includes memory including computer readable instructions. The example apparatus also includes a processor to execute the computer readable instructions to generate a media request message to obtain media hosted by a server, insert, into the media request message, a source indicator of the media requesting source, the source indicator indicating whether the media requesting source is associated with a media provider to provide audience measurement information associated with the media provided by the media provider, transmit the media request message to the server, and execute a delay tag in a media response message received from the server, the delay tag to cause a delay time period based on the source indicator.
1. An apparatus comprising:
memory including computer readable instructions; and
a processor to execute the computer readable instructions to:
generate a media request message to obtain media hosted by a server;
insert, into the media request message, a source indicator of a media requesting source, the source indicator indicating whether the media requesting source is associated with a media provider, the source indicator to indicate audience measurement information associated with the media provided by the media provider;
transmit the media request message to the server; and
execute a delay tag in a media response message received from the server, the delay tag to cause a delay time period based on the source indicator.
2. The apparatus as defined in claim 1 , further including a web crawler.
3. The apparatus as defined in claim 1 , wherein the delay tag includes executable instructions.
4. The apparatus as defined in claim 3 , wherein the executable instructions include a native delay function.
5. The apparatus as defined in claim 1 , wherein execution of the delay tag generates the delay time period.
6. The apparatus as defined in claim 1 , wherein execution of the delay tag requests data from a web site associated with generating the delay time period.
7. The apparatus as defined in claim 1 , wherein executing the delay time period in the media response message includes selecting a pre-existing copy of the media with a second delay tag associated with the delay time period.
8. The apparatus as defined in claim 1 , wherein the media request message is either a hypertext transfer protocol GET request or a hypertext transfer protocol POST request.
9. A non-transitory tangible computer readable storage medium comprising instructions that, when executed, cause a processor to at least:
generate a media request message to obtain media hosted by a server;
insert, into the media request message, a source indicator of a media requesting source, the source indicator indicating whether the media requesting source is associated with a media provider, the source indicator to indicate audience measurement information associated with the media provided by the media provider;
transmit the media request message to the server; and
execute a delay tag in a media response message received from the server, the delay tag to cause a delay time period based on the source indicator.
10. The computer readable storage medium as defined in claim 9 , wherein executing the delay tag causes the processor to request data from a web site associated with generating the delay time period.
11. The computer readable storage medium as defined in claim 9 , wherein executing the delay tag in the media response message includes selecting a pre-existing copy of the media with a second delay tag associated with the delay time period.
12. The computer readable storage medium as defined in claim 9 , wherein the media request message is either a hypertext transfer protocol GET request or a hypertext transfer protocol POST request.
13. The computer readable storage medium as defined in claim 9 , wherein the delay tag includes executable instructions, the executable instructions including a native delay function.
14. A method comprising:
generating, by executing an instruction with a processor, a media request message to obtain media hosted by a server;
inserting, by executing an instruction with the processor, into the media request message, a source indicator of a media requesting source, the source indicator indicating whether the media requesting source is associated with a media provider, the source indicator to indicate audience measurement information associated with the media provided by the media provider;
transmitting, by executing an instruction with the processor, the media request message to the server; and
executing a delay tag in a media response message received from the server, the delay tag to cause a delay time period based on the source indicator.
15. The method as defined in claim 14 , wherein the delay tag includes executable instructions.
16. The method as defined in claim 15 , wherein the executable instructions include a native delay function.
17. The method as defined in claim 14 , wherein execution of the delay tag generates the delay time period.
18. The method as defined in claim 14 , wherein execution of the delay tag requests data from a web site associated with generating the delay time period.
19. The method as defined in claim 14 , wherein inserting the time delay in the media response message includes selecting a pre-existing copy of the media with a second delay tag associated with the delay time period.
20. The method as defined in claim 14 , wherein the media request message is either a hypertext transfer protocol GET request or a hypertext transfer protocol POST request.