Methods and apparatus to supplement web crawling with cached data from distributed devices
View Patent ↗Methods and apparatus to supplement web crawling with cached data from distributed devices are disclosed. An example method includes accessing a first set of websites cached in a panelist device; comparing the first set of websites to a second set of websites to be analyzed by a crawler; and retrieving with the crawler a first website included in the second set of websites but not included in the first set of websites from a server associated with the first website.
1. A method comprising:
identifying a first webpage;
accessing, by executing an instruction with a processor, a set of cached webpages cached in a panelist device;
when the first webpage is in the set of cached webpages:
determining, by executing an instruction with the processor, a first physical location from which the first webpage was requested by the panelist device; and
transmitting, by executing an instruction with the processor, a plurality of requests for the first webpage, the requests in the plurality of requests identifying respective ones of a plurality of physical locations, the requests in the plurality of requests skipping the first physical location from which the first webpage was requested by the panelist device, thereby reducing the number of requests sent to the first webpage.
2. The method as defined in claim 1 , wherein the set of cached webpages is accessed from a browser history of the panelist device.
3. The method as defined in claim 2 , wherein the browser history is native to a browser of the panelist device.
4. The method as defined in claim 1 , wherein the set of cached webpages is accessed from a log of webpages in the panelist device.
5. The method as defined in claim 1 , when the first webpage is in the set of cached webpages further including:
identifying a first browser type used by the panelist device to access the first webpage; and
transmitting, by executing an instruction with the processor, a second plurality of requests for the first webpage, the requests in the second plurality identifying different browser types, the requests in the second plurality of requests skipping the first browser type used by the panelist device to access the first webpage, thereby reducing the number of requests sent to the first webpage.
6. The method as defined in claim 1 , when the first webpage is in the set of cached webpages further including:
identifying a first language setting used by the panelist device when the first webpage was accessed; and
transmitting, by executing an instruction with the processor, a second plurality of requests for the first webpage, respective ones of the requests in the second plurality identifying different language settings, the requests in the second plurality of requests skipping the first language setting used by the panelist device when the first webpage was accessed, thereby reducing the number of requests sent to the first webpage.
7. The method as defined in claim 1 , wherein the plurality of requests for the first webpage are transmitted from the processor via a monitoring data collection site.
8. A tangible machine-readable storage medium comprising instructions which, when executed, cause a machine to at least:
identify a first webpage;
access a set of cached webpages cached in a client device;
when the first webpage is in the set of cached webpages:
determine, with a processor, a first physical location from which the first webpage was requested by the client device; and
transmit a plurality of requests for the first webpage, the requests in the plurality of requests identifying respective ones of a plurality of physical locations, the requests in the plurality of requests skipping the first physical location from which the first webpage was requested by the client device, thereby reducing the number of requests sent to the first webpage.
9. The tangible machine-readable storage medium as defined in claim 8 , wherein the instructions cause the machine to access the set of cached webpages from a browser history of the client device.
10. The tangible machine-readable storage medium as defined in claim 9 , wherein the browser history is stored by a browser of the client device.
11. The tangible machine-readable storage medium as defined in claim 8 , wherein the instructions cause the machine to access the set of cached webpages from a log of webpages in the client device.
12. The tangible machine-readable storage medium as defined in claim 8 , further including instructions which, when executed, cause the machine to at least:
when the first webpage is in the set of cached webpages:
identify a first browser type used by the panelist device to access the first webpage; and
transmit a second plurality of requests for the first webpage, respective ones of the requests in the second plurality identifying different browser types, the requests in the second plurality of requests skipping the first browser type used by the panelist device to access the first webpage, thereby reducing the number of requests sent to the first webpage.
13. The tangible machine-readable storage medium as defined in claim 8 , further including instructions which, when executed, cause the machine to at least:
when the first webpage is in the set of cached webpages:
identify a first language setting used by the panelist device when the first webpage was accessed; and
transmit a second plurality of requests for the first webpage, respective ones of the requests in the second plurality identifying language settings, the requests in the second plurality of requests skipping the first language setting used by the panelist device when the first webpage was accessed, thereby reducing the number of requests sent to the first webpage.
14. The tangible machine-readable storage medium as defined in claim 8 , wherein the requests are transmitted for the first webpage via the client device.
15. An apparatus to supplement web crawling with cached data from a distributed device, the apparatus comprising:
a data receiver to identify a webpage of a set of cached webpages cached in the distributed device; and
a crawler to identify a first physical location from which the webpage was requested by the distributed device, a web requester to transmit a plurality of requests for the webpage, the requests in the plurality of requests identifying respective ones of a plurality of physical locations as the source of the request, the requests in the plurality of requests skipping the first physical location from which the webpage was requested by the distributed device, thereby reducing the number of requests sent to the webpage.
16. The apparatus as defined in claim 15 , wherein:
the crawler is to access a first browser type used by the distributed device to access the webpage; and
the web requester is to transmit a second plurality of requests for the webpage, respective ones of the second plurality of requests identifying different browser types, the second plurality of requests skipping the first browser type from which the first webpage was accessed by the distributed device, thereby reducing the number of requests sent to the webpage.
17. The apparatus as defined in claim 15 , wherein:
the crawler is to access a first language setting used by the distributed device when the first webpage was accessed; and
the web requester is to transmit a second plurality of requests for the webpage, respective ones of the second plurality of requests identifying different language settings, the second plurality of requests skipping the first language setting used by the distributed device when the first webpage was accessed, thereby reducing the number of requests sent to the webpage.
18. The apparatus as defined in claim 15 , wherein the web requester is to transmit the plurality of requests for the webpage via the distributed device.