Methods and apparatus to supplement web crawling with cached data from distributed devices
Methods and apparatus to supplement web crawling with cached data from distributed devices are disclosed. An example apparatus includes a browser monitor to extract a language setting from a web browser of a panelist computing device, and a data communicator to transmit data associated with a webpage accessed by the web browser and an indication of the language setting to a server.
1. An apparatus comprising:
a browser monitor to extract a language setting from a first web browser type of a panelist computing device; and
a data communicator to transmit data associated with a webpage accessed by the first web browser type and an indication of the language setting to an audience measurement entity server, the audience entity measurement server to generate a number of web requests for the webpage from a web server hosting the webpage, the web requests indicative of a request to retrieve the webpage, respective ones of the web requests identifying different language settings than the extracted language setting, the respective ones of the web requests identifying different web browser types than the first web browser type.
2. The apparatus of claim 1 , further including a cache reader to identify the webpage by accessing a set of cached webpages cached in the panelist computing device.
3. The apparatus of claim 2 , wherein the set of cached webpages is accessed from a log of webpages in the panelist computing device.
4. The apparatus of claim 1 , wherein the browser monitor is to determine that the webpage has not been transmitted to the audience measurement entity server, and, in response to determining that the webpage has not been transmitted to the audience measurement entity server, the data communicator is to transmit the data associated with the webpage to the audience measurement entity server.
5. The apparatus of claim 1 , wherein the audience measurement entity server is to reduce a total number of requests sent to the web server in response to obtaining the indication of the language setting from the data communicator, the audience measurement entity server to generate the number of requests based on the indication of the language setting, respective ones of the requests in the number identifying different language settings than the language setting used by the panelist computing device when the webpage was accessed.
6. The apparatus of claim 1 , wherein the data communicator is to skip the first web browser type used by the panelist computing device to access the webpage, thereby reducing a total number of requests sent to the webpage by the audience measurement entity server.
7. The apparatus of claim 1 , wherein the number of requests is a first number of requests, the browser monitor to identify a first physical location of the panelist computing device when the webpage was accessed, the data communicator to transmit the first physical location to the audience measurement entity server, and the audience measurement entity server to transmit a second number of requests for the webpage, respective ones of the requests in the second number identifying different physical locations than the first physical location, the requests in the second number of requests skipping the first physical location from which the webpage was requested by the panelist computing device, thereby reducing a total number of requests sent to the webpage by the audience measurement entity server.
8. The apparatus of claim 1 , wherein the browser monitor is to store the data in a data store by accessing a browser history of the panelist computing device, the data including at least one of the language setting, a uniform resource locator, content of the webpage, a type of the first web browser type that accessed the webpage, a time and date when the panelist computing device accessed the webpage, or a location of the panelist computing device when the webpage was accessed.
9. The apparatus of claim 8 , wherein the browser history is native to the first web browser type of the panelist computing device, and the browser monitor is to generate a log including the data in response to the first web browser type accessing the webpage, the log different from the browser history and a set of cached webpages cached in the panelist computing device.
10. A non-transitory computer readable storage medium comprising instructions that, when executed, cause at least one processor to:
extract a language setting from a first web browser type of a panelist computing device; and
transmit data associated with a webpage accessed by the first web browser type and an indication of the language setting to audience measurement entity server, the audience measurement entity server to generate a number of web requests for the webpage from a web server hosting the webpage, the web requests indicative of a request to retrieve the webpage, respective ones of the web requests identifying different language settings than the extracted language setting, the respective ones of the web requests identifying different web browser types than the first web browser type.
11. The non-transitory computer readable storage medium of claim 10 , wherein the audience measurement entity server is to reduce a total number of requests sent to the web server in response to obtaining the indication of the language setting from the panelist computing device, the audience measurement entity server to generate the number of requests based on the indication of the language setting, respective ones of the requests identifying different language settings than the language setting used by the panelist computing device when the webpage was accessed.
12. The non-transitory computer readable storage medium of claim 10 , wherein the number of requests is a first number of requests, and the instructions, when executed, cause the at least one processor to:
skip the first web browser type used by the panelist computing device to access the webpage, thereby reducing a total number of requests sent to the webpage by the audience measurement entity server.
13. The non-transitory computer readable storage medium of claim 10 , wherein the number of requests is a first number of requests, and the instructions, when executed, cause the at least one processor to:
identify a first physical location of the panelist computing device when the webpage was accessed; and
transmit the first physical location to the audience measurement entity server, the audience measurement entity server to transmit a second number of requests for the webpage, respective ones of the requests in the second number identifying different physical locations than the first physical location, the requests in the second number of requests skipping the first physical location from which the webpage was requested by the panelist computing device, thereby reducing a total number of requests sent to the webpage by the audience measurement entity server.
14. The non-transitory computer readable storage medium of claim 13 , wherein the instructions, when executed, cause the at least one processor to:
store the data in a data store by accessing a browser history of the panelist computing device, the browser history is native to the first web browser type of the panelist computing device; and
generate a log including the data in response to the first web browser type accessing the webpage, the log different from the browser history and a set of webpages cached in the panelist computing device.
15. A system comprising:
a panelist computing device to:
extract a language setting from a first web browser type; and
transmit data associated with a webpage accessed by the first web browser type and an indication of the language setting to an audience measurement entity server; and
the audience measurement entity server to transmit a number of web requests for the webpage from a web server hosting the webpage, the web requests indicative of a request to retrieve the webpage, respective ones of the web requests identifying different language settings than the language setting, the respective ones of the web requests identifying different web browser types than the first web browser type.
16. The system of claim 15 , wherein the audience measurement entity server is to reduce a total number of requests sent to the web server by generating the number of requests to skip the language setting used by the panelist computing device when the webpage was accessed.
17. The system of claim 15 , wherein the panelist computing device is to skip the first web browser type used to access the webpage, thereby reducing a total number of requests sent to the webpage by the audience measurement entity server.
18. The system of claim 15 , wherein the number of requests is a first number of requests, and the panelist computing device is to identify a first physical location of the panelist computing device when the webpage was accessed and transmit the first physical location to the audience measurement entity server, and the audience measurement entity server is to transmit a second number of requests for the webpage, respective ones of the requests in the second number identifying different physical locations than the first physical location, the requests in the second number of requests skipping the first physical location from which the webpage was requested by the panelist computing device, thereby reducing a total number of requests sent to the webpage by the audience measurement entity server.
19. The system of claim 15 , wherein the panelist computing device is to store the data in a data store by accessing a browser history of the panelist computing device, the data including at least one of the language setting, a uniform resource locator, content of the webpage, a type of the first web browser type that accessed the webpage, a time and date when the panelist computing device accessed the webpage, or a location of the panelist computing device when the webpage was accessed.
20. The system of claim 19 , wherein the browser history is native to the first web browser type of the panelist computing device, and the panelist computing device is to generate a log including the data in response to the first web browser type accessing the webpage, the log different from the browser history and a set of webpages cached in the panelist computing device.
21. An apparatus comprising:
memory;
instructions in the apparatus; and
processor circuitry to execute the instructions to:
extract a language setting from a first web browser type of a panelist computing device; and
transmit data associated with a webpage accessed by the first web browser type and an indication of the language setting to audience measurement entity server, the audience measurement entity server to generate a number of web requests for the webpage from a web server hosting the webpage, the web requests indicative of a request to retrieve the webpage, respective ones of the web requests identifying different language settings than the extracted language setting, the respective ones of the web requests identifying different web browser types than the first web browser type.
22. The apparatus of claim 21 , wherein the audience measurement entity server is to reduce a total number of requests sent to the web server in response to obtaining the indication of the language setting from the panelist computing device, the audience measurement entity server to generate the number of requests based on the indication of the language setting, respective ones of the requests identifying different language settings than the language setting used by the panelist computing device when the webpage was accessed.
23. The apparatus of claim 21 , wherein the number of requests is a first number of requests, and the instructions, when executed, cause the processor circuitry to:
skip the first web browser type used by the panelist computing device to access the webpage, thereby reducing a total number of requests sent to the webpage by the audience measurement entity server.
24. The apparatus of claim 21 , wherein the number of requests is a first number of requests, and the instructions, when executed, cause the processor circuitry to:
identify a first physical location of the panelist computing device when the webpage was accessed; and
transmit the first physical location to the audience measurement entity server, the audience measurement entity server to transmit a second number of requests for the webpage, respective ones of the requests in the second number identifying different physical locations than the first physical location, the requests in the second number of requests skipping the first physical location from which the webpage was requested by the panelist computing device, thereby reducing a total number of requests sent to the webpage by the audience measurement entity server.
25. The apparatus of claim 24 , wherein the instructions, when executed, cause the processor circuitry to:
store the data in a data store by accessing a browser history of the panelist computing device, the browser history is native to the first web browser type of the panelist computing device; and
generate a log including the data in response to the first web browser type accessing the webpage, the log different from the browser history and a set of webpages cached in the panelist computing device.