Methods and apparatus to supplement web crawling with cached data from distributed devices
Methods and apparatus to supplement web crawling with cached data from distributed devices are disclosed. An example method includes identifying a first webpage. Accessing a set of cached webpages cached in a panelist device. When the first webpage is in the set of cached webpages: identifying a first language setting used by the panelist device when the first webpage was accessed. Transmitting a first number of requests for the first webpage, respective ones of the requests in the first number identifying different language settings. The requests in the first number of requests skipping the first language setting used by the panelist device when the first webpage was accessed.
1. A method comprising:
identifying a first webpage;
accessing, by executing an instruction with a processor, a set of cached webpages cached in a panelist device; and
when the first webpage is in the set of cached webpages:
identifying, by executing an instruction with the processor, a first language setting used by the panelist device when the first webpage was accessed; and
transmitting, by executing an instruction with the processor, a first number of requests for the first webpage, respective ones of the requests in the first number identifying different language settings, the requests in the first number of requests skipping the first language setting used by the panelist device when the first webpage was accessed, thereby reducing a total number of requests sent to the first webpage.
2. The method as defined in claim 1 , wherein the set of cached webpages is accessed from a browser history of the panelist device.
3. The method as defined in claim 2 , wherein the browser history is native to a browser of the panelist device.
4. The method as defined in claim 1 , wherein the set of cached webpages is accessed from a log of webpages in the panelist device.
5. The method as defined in claim 1 , when the first webpage is in the set of cached webpages further including:
identifying a first browser type used by the panelist device to access the first webpage; and
transmitting, by executing an instruction with the processor, a second number of requests for the first webpage, respective ones of the requests in the second number identifying different browser types, the requests in the second number of requests skipping the first browser type used by the panelist device to access the first webpage, thereby reducing the total number of requests sent to the first webpage.
6. The method as defined in claim 5 , when the first webpage is in the set of cached webpages further including:
identifying a first physical location from which the first webpage was requested by the panelist device; and
transmitting, by executing an instruction with the processor, a third number of requests for the first webpage, the requests in the third number of requests identifying respective ones of a plurality of physical locations, the requests in the third number of requests skipping the first physical location from which the first webpage was requested by the panelist device, thereby reducing the total number of requests sent to the first webpage.
7. The method as defined in claim 1 , wherein the first number of requests for the first webpage are transmitted from the processor via a monitoring data collection site.
8. The method of claim 1 , further including obtaining first cache data associated with the first webpage from an on-device meter operating on the panelist device, the on-device meter to transmit the first cache data to a server associated with the processor by:
monitoring a browser operating on the panelist device;
in response to the browser visiting the first webpage, generating the first cache data by caching the first webpage and a uniform resource locator (URL) identifying the first webpage; and
transmitting the first cache data to the server, the server to identify the first webpage based on the URL.
9. A tangible machine-readable storage medium comprising instructions which, when executed, cause a machine to at least:
identify a first webpage;
access a set of cached webpages cached in a client device;
when the first webpage is in the set of cached webpages:
identify a first language setting used by the client device when the first webpage was accessed; and
transmit a number of requests for the first webpage, respective ones of the requests in the number identifying different language settings, the requests in the number of requests skipping the first language setting used by the client device when the first webpage was accessed, thereby reducing a total number of requests sent to the first webpage.
10. The tangible machine-readable storage medium as defined in claim 9 , wherein the instructions cause the machine to access the set of cached webpages from a browser history of the client device.
11. The tangible machine-readable storage medium as defined in claim 10 , wherein the browser history is stored by a browser of the client device.
12. The tangible machine-readable storage medium as defined in claim 9 , wherein the instructions cause the machine to access the set of cached webpages from a log of webpages in the client device.
13. The tangible machine-readable storage medium as defined in claim 9 , further including instructions which, when executed, cause the machine to at least:
when the first webpage is in the set of cached webpages:
identify a first browser type used by the client device to access the first webpage; and
transmit a second number of requests for the first webpage, respective ones of the requests in the second number identifying different browser types, the requests in the second number of requests skipping the first browser type used by the client device to access the first webpage, thereby reducing the total number of requests sent to the first webpage.
14. The tangible machine-readable storage medium as defined in claim 13 , further including instructions which, when executed, cause the machine to at least:
when the first webpage is in the set of cached webpages:
determine a first physical location from which the first webpage was requested by the client device; and
transmit a third number of requests for the first webpage, the requests in the third number of requests identifying respective ones of a plurality of physical locations, the requests in the third number of requests skipping the first physical location from which the first webpage was requested by the client device, thereby reducing the total number of requests sent to the first webpage.
15. The tangible machine-readable storage medium as defined in claim 9 , wherein the requests are transmitted for the first webpage via the client device.
16. The tangible machine-readable storage medium of claim 9 , wherein the instructions, when executed, cause the machine to obtain first cache data associated with the first webpage from an on-device meter operating on the client device, the on-device meter is to transmit the first cache data to the machine by:
monitoring a browser operating on the client device;
in response to the browser visiting the first webpage, generating the first cache data by caching the first webpage and a uniform resource locator (URL) identifying the first webpage; and
transmitting the first cache data to the machine, the machine to identify the first webpage based on the URL.
17. An apparatus to supplement web crawling with cached data from a distributed device, the apparatus comprising:
a data receiver to identify a webpage of a set of cached webpages cached in the distributed device; and
a crawler to access a first language setting used by the distributed device when the webpage was accessed; and
a web requester to transmit a number of requests for the webpage, the requests in the number of requests identifying different language settings, the number of requests skipping the first language setting used by the distributed device when the webpage was accessed, thereby reducing a total number of requests sent to the webpage.
18. The apparatus as defined in claim 17 , wherein:
the crawler is to access a first browser type used by the distributed device to access the webpage; and
the web requester is to transmit a second number of requests for the webpage, respective ones of the second number of requests identifying different browser types, the second number of requests skipping the first browser type from which the webpage was accessed by the distributed device, thereby reducing the total number of requests sent to the webpage.
19. The apparatus as defined in claim 18 , wherein:
the crawler is to identify a first physical location from which the webpage was requested by the distributed device; and
the web requester is to transmit a third number of requests for the webpage, the requests in the third number of requests identifying respective ones of a plurality of physical locations as a source of the requests, the requests in the third number of requests skipping the first physical location from which the webpage was requested by the distributed device, thereby reducing the total number of requests sent to the webpage.
20. The apparatus as defined in claim 17 , wherein the web requester is to transmit the number of requests for the webpage via the distributed device.
21. The apparatus as defined in claim 17 , wherein the set of cached webpages is accessed from a browser history of the distributed device.
22. The apparatus as defined in claim 17 , wherein the set of cached webpages is accessed from a log of webpages in the distributed device.
23. The apparatus of claim 17 , wherein the data receiver is to obtain cache data associated with the webpage from an on-device meter operating on the distributed device, the on-device meter is to transmit the cache data to the data receiver by:
monitoring a browser operating on the distributed device;
in response to the browser visiting the webpage, generating the cache data by caching the webpage and a uniform resource locator (URL) identifying the webpage; and
transmitting the cache data to the data receiver, the data receiver to identify the webpage based on the URL.