Hint model updating using automated browsing clusters
Embodiments seek to improve prefetch hinting by using automated browsing clusters to generate and update hinting models used for machine-generated hints. For example, hinting machines can include browsing clusters that autonomously fetch web pages in response to update triggers (e.g., client web page requests, scheduled web crawling, etc.) and generate timing and/or other hinting-related feedback relating to which resources were used to load the fetched web pages. The hinting machines can use the hinting feedback to generate and/or update hinting models, which can be used for machine-generation of hints. Some embodiments can provide preliminary hinting functionality in response to client hinting requests, for example, when hinting models for a requested page are insufficient (e.g., unavailable, outdated, etc.). For example, without having a sufficient hinting model in place, the hinting machine can fetch the page to generate preliminary hinting feedback, which it can use to machine-generate preliminary hints.
1. A method for hint model updating using an automated browsing cluster in communication with a machine-driven hinting server, the method comprising:
determining, by the automated browsing cluster, a condition for updating a hinting model associated with a web page has been satisfied, the hinting model maintained at the machine-driven hinting server to inform remotely located web browsers of resources to prefetch as part of a future rendering of the web page by any of the web browsers;
fetching the web page, by the automated browsing cluster, in response to the determining, wherein the automated browsing cluster is located remotely from the web browsers;
identifying, by the automated browsing cluster from the fetched web page, page fetching data for resources of the web page;
generating hinting feedback, by the automated browsing cluster, according to the page fetching data; and
communicating the hinting feedback from the automated browsing cluster to the machine-driven hinting server for updating of the hinting model by the machine-driven hinting server.
2. The method of claim 1 , wherein the condition comprises a load of the web page.
3. The method of claim 1 , wherein the condition comprises a threshold number of loads of the web page.
4. The method of claim 1 wherein the condition comprises a predetermined time period has elapsed.
5. The method of claim 1 , wherein the condition comprises a predetermined scheduled time.
6. The method of claim 1 , wherein:
the automated browsing cluster is disposed on the machine-driven hinting server;
the hinting server is separated from each of the web browsers by a communications network; and
the determining is performed by the machine-driven hinting server.
7. The method of claim 1 further comprising updating, by the machine-driven hinting server, the hinting model using the hinting feedback.
8. The method of claim 7 , further comprising:
receiving a hinting request from one of the web browsers for hints rendering the web page; and
communicating a model-based hinting response from the machine-driven hinting server to the requesting web browser in response to the hinting request, the model-based hinting response generated according to the updated hinting model.
9. The method of claim 8 , further comprising:
determining the condition, by the machine-driven hinting server and in response to the hinting request, based on there not being a hinting model for the web page at the machine-driven hinting server;
generating, by the machine-driven hinting server, a preliminary hinting response from the hinting feedback received from the automated browsing cluster;
communicating the preliminary hinting response from the machine-driven hinting server to the requesting web browser substantially in real time in response to the hinting request; and
updating the hinting model by creating the hinting model on the machine-driven hinting server from the hinting feedback.
10. The method of claim 1 , wherein the condition corresponds to a determination that the hinting model for the web page at the machine-driven hinting server is at least one of incomplete or outdated.
11. The method of claim 1 further comprising:
rendering, by the automated browsing cluster, the web page fetched by the automated browsing cluster; and
wherein the page fetching data comprises page rendering data from the rendering of the web page by the automated browsing cluster.
12. The method of claim 11 , wherein the page rendering data further comprises identifications of child resources of the web page fetched as part of the rendering of the web page by the automated browsing cluster.
13. The method of claim 1 , wherein:
the fetching comprises fetching, by the automated browsing cluster, the web page a plurality of times,
the identifying comprises identifying, by the automated browsing cluster, page fetching data from each of the fetchings of the web page, and
the generating comprises generating the hinting feedback, by the automated browsing cluster, according to the page fetching data identified from each of the fetchings of the web page.
14. The method of claim 1 , wherein:
the automated browsing cluster comprises a plurality of web page fetching scripts, each web page fetching scripts having less than a full functionality of any of the web browsers; and
the fetching comprises one of the web page fetching scripts fetching the web page.
15. The method of claim 1 , wherein:
the automated browsing cluster emulates a plurality of different web page fetching environments;
the fetching comprises a first fetching of the web page in a first of the web fetching environments and a second fetching of the web page in a second of the web fetching environments, wherein the second web fetching environment is different than the first web fetching environment;
the identifying comprises identifying, by the automated browsing cluster, page fetching data form both the first fetching and the second fetching of the web page; and
the generating comprises generating the hinting feedback, by the automated browsing cluster, according to the page fetching data identified from both the first fetching and the second fetching of the web page.
16. A machine-driven hint generation system comprising:
a data store comprising a hinting model for a web page, wherein the hinting model comprises hint information usable by remotely located web browsers of resources to prefetch as part of a rendering of the web page by any of the web browsers;
a hinting server, in communication with the data store, to respond to hint requests for the web page from any of the web browsers using the hinting model; and
an automated browsing cluster, in communication with the hinting server, that operates, based on a determination that a condition for updating a hinting model associated with the web page has been satisfied to:
fetch the web page;
identify, from the fetched web page, page fetching data for resources of the web page; and
generate hinting feedback according to the page fetching data;
wherein the data store, the hinting server, and the automated browsing cluster are located remotely from the web browsers; and
wherein the automated browsing cluster in communication with the hinting server is configured to determining the condition for updating the hinting model associated with the web page has been satisfied.
17. The system of claim 16 , wherein the condition comprises a load of the web page.
18. The system of claim 16 , wherein the condition comprises a threshold number of loads of the web page.
19. The system of claim 16 , wherein the condition comprises a predetermined time period has elapsed.
20. The system of claim 16 , wherein the condition comprises a predetermined scheduled time.
21. The system of claim 16 , wherein:
the automated browsing cluster is disposed on the hinting server; and
the hinting server is separated from the web browsers by a communications network.
22. The system of claim 16 , wherein the hinting server further operates to:
receive a hinting request from one of the web browsers for hints for rendering the web page;
generate according to the updated hinting model a model-based hinting response; and
communicate the model-based hinting response to the requesting web browser.
23. The system of claim 16 , wherein the hinting server further operates to:
upon determining that the hinting model for the web page is not stored in the data store, determine the condition for updating the hinting model associated with the web page has been satisfied;
generate a preliminary hinting response from the hinting feedback;
communicate the preliminary hinting response to the requesting web browser substantially in real time in response to the hinting request; and
create the hinting model from the hinting feedback.
24. The system of claim 16 , wherein the hinting server further operates to determine the condition for updating the hinting model associated with the web page has been satisfied upon determining that the hinting model in the data store for the web page is at least one of incomplete or outdated.
25. The system of claim 16 , wherein:
the automated browsing cluster further operates to render the fetched web page, and
the page fetching data comprises page rendering data from the rendering of the web page by the automated browsing cluster.
26. The system of claim 25 , wherein the page rendering data further comprises identifications of child resources of the web page fetched by the automated browsing cluster as part of rendering the web page.
27. The system of claim 26 , wherein the page rendering data further comprises load timing information for a plurality of child resources.
28. The system of claim 16 , wherein the automated browsing cluster operates further to:
fetch the web page a plurality of times,
identify page fetching data from each of the fetchings of the web page, and
generate the hinting feedback according to the page fetching data identified from each of the fetchings of the web page.
29. The system of claim 28 , wherein the automated browsing cluster operates further to fetch the web page the plurality of times substantially in parallel.
30. The system of claim 16 , wherein the automated browsing cluster comprises a web page fetching script having less than a full functionality of any of the web browsers, and the automated browsing cluster operates to fetch the web page using the web page fetching script.