Hint model updating using automated browsing clusters
Embodiments seek to improve prefetch hinting by using automated browsing clusters to generate and update hinting models used for machine-generated hints. For example, hinting machines can include browsing clusters that autonomously fetch web pages in response to update triggers (e.g., client web page requests, scheduled web crawling, etc.) and generate timing and/or other hinting-related feedback relating to which resources were used to load the fetched web pages. The hinting machines can use the hinting feedback to generate and/or update hinting models, which can be used for machine-generation of hints. Some embodiments can provide preliminary hinting functionality in response to client hinting requests, for example, when hinting models for a requested page are insufficient (e.g., unavailable, outdated, etc.). For example, without having a sufficient hinting model in place, the hinting machine can fetch the page to generate preliminary hinting feedback, which it can use to machine-generate preliminary hints.
1. A method for hint model updating using an automated browsing cluster in communication with a machine-driven hinting server, the method comprising:
issuing, by the hinting server to the automated browsing cluster, an update trigger for updating a hinting model associated with a web page, the hinting model maintained at the hinting server for informing remotely located web browsers on remotely located client machines of resources to prefetch as part of a future rendering of the web page by any of the web browsers;
fetching the web page, by the automated browsing cluster, in response to the update trigger issued by the hinting server, wherein the automated browsing cluster is located remotely from the web browsers of the client machines;
identifying, by the automated browsing cluster from the fetched web page, page fetching data for resources of the web page;
generating hinting feedback, by the automated browsing cluster, according to the page fetching data; and
communicating the hinting feedback from the automated browsing cluster to the hinting server for updating of the hinting model by the hinting server.
2. The method of claim 1 , wherein:
the automated browsing cluster is disposed on the hinting server;
the hinting server is separated from each of the web browsers by a communications network;
the issuing comprises a module of the hinting server issuing the update trigger to the automated browsing cluster; and
the communicating comprises the automated browsing cluster communicating the hinting feedback to the module of the hinting server.
3. The method of claim 1 further comprising updating, by the hinting server, the hinting model using the hinting feedback.
4. The method of claim 3 , further comprising:
receiving a hinting request from one of the web browsers for hints for rendering the web page; and
communicating a model-based hinting response from the hinting server to the requesting browser in response to the hinting request, the model-based hinting response generated according to the updated hinting model.
5. The method of claim 1 , wherein the update trigger corresponds to receipt of a hinting request at the hinting server from one of the web browsers for hints for rendering the web page.
6. The method of claim 5 , further comprising:
determining, by the hinting server, in response to the hinting request, that there is not a hinting model for the web page at the hinting server, wherein the hinting server issues the update trigger in response to the determining that there is not a hinting model for the web page at the hinting server;
generating, by the hinting server, a preliminary hinting response from the hinting feedback received from the automated browsing cluster;
communicating the preliminary hinting response from the hinting server to the requesting web browser substantially in real time in response to the hinting request; and
updating the hinting model by creating the hinting model on the hinting server from the hinting feedback.
7. The method of claim 1 , wherein the update trigger corresponds to a determination that the hinting model for the web page at the hinting server is incomplete and/or outdated.
8. The method of claim 1 , wherein at least one of the resources identified by the automated browsing cluster is a uniform resource locator (URL) of a child resource of the web page.
9. The method of claim 1 further comprising:
rendering, by the automated browsing cluster, the web page fetched by the automated browsing cluster,
wherein the page fetching data comprises page rendering data from the rendering of the web page by the automated browsing cluster.
10. The method of claim 9 , wherein the page rendering data further comprises identifications of child resources of the web page fetched as part of the rendering of the web page by the automated browsing cluster.
11. The method of claim 10 , wherein the page rendering data further comprises load timing information for a plurality of the child resources.
12. The method of claim 1 , wherein:
the fetching comprises fetching, by the automated browsing cluster, the web page a plurality of times,
the identifying comprises identifying, by the automated browsing cluster, page fetching data from each of the fetchings of the web page, and
the generating comprises generating the hinting feedback, by the automated browsing cluster, according to the page fetching data identified from each of the fetchings of the web page.
13. The method of claim 12 , wherein the fetching further comprises fetching the web page the plurality of times substantially in parallel.
14. The method of claim 1 , wherein;
the automated browsing cluster comprises a plurality of web page fetching scripts, each web page fetching script having less than a full functionality of any of the web browsers on the client machines, and
the fetching comprises one of the web page fetching scripts fetching the web page.
15. The method of claim 1 , wherein:
the automated browsing cluster emulates a plurality of different web page fetching environments;
the fetching comprises a first fetching of the web page in a first of the web fetching environments and a second fetching of the web page in a second of the web fetching environments, wherein the second web fetching environment is different than the first web fetching environment;
the identifying comprises identifying, by the automated browsing cluster, page fetching data from both the first fetching and the second fetching of the web page; and
the generating comprises generating the hinting feedback, by the automated browsing cluster, according to the page fetching data identified from both the first fetching and the second fetching of the web page.
16. The method of claim 1 , further comprising:
scheduling the fetching of the web page by the automated browsing cluster during an off-peak network usage timeframe,
wherein the fetching is according to the scheduling.
17. A machine-driven hint generation system comprising:
a data store that maintains a hinting model for a web page, wherein the hinting model is for informing remotely located web browsers of resources to prefetch as part of a rendering of the web page by any of the web browsers;
a hinting server, in communication with the data store, that operates to issue an update trigger in response to determining to update the hinting model; and
an automated browsing cluster, in communication with the hinting server, that operates, in response to the update trigger from the hinting server to:
fetch the web page;
identify, from the fetched web page, page fetching data for resources of the web page; and
generate hinting feedback according to the page fetching data;
wherein the data store, the hinting server, and the automated browsing cluster are located remotely from the web browsers; and
wherein the hinting server issues the update trigger to the automated browsing cluster.
18. The system of claim 17 , wherein:
the automated browsing cluster is disposed on the hinting server,
a module of the hinting server is configured to issue the update trigger to the automated browsing cluster, and
the hinting server is separated from the web browsers by a communications network.
19. The system of claim 17 , wherein:
the automated browsing cluster further operates to communicate the hinting feedback to the hinting server; and
the hinting server further operates to update the hinting model according to the hinting feedback.
20. The system of claim 19 , wherein the hinting server further operates to:
receive a hinting request from one of the web browsers for hints for rendering the web page;
generate according to the updated hinting model a model-based hinting response; and
communicate the model-based hinting response to the requesting browser.
21. The system of claim 17 , wherein the hinting server further operates to receive a hinting request from one of the browsers for hints for rendering the web page.
22. The system of claim 21 , wherein the hinting server further operates to:
upon determining that the hinting model for the web page is not stored in the data store, issue the update trigger;
generate a preliminary hinting response from the hinting feedback; and
communicate the preliminary hinting response to the requesting web browser substantially in real time in response to the hinting request; and
create the hinting model from the hinting feedback.
23. The system of claim 17 , wherein the hinting server further operates to issue the update trigger upon determining that the hinting model in the data store for the web page is incomplete and/or outdated.
24. The system of claim 17 , wherein at least one of the resources for which the automated browsing cluster identifies page fetching data is a uniform resource locator (URL) of a child resource of the web page.
25. The system of claim 17 , wherein:
the automated browsing cluster further operates to render the fetched web page, and
the page fetching data comprises page rendering data from the rendering of the web page by the automated browsing cluster.
26. The system of claim 25 , wherein the page rendering data further comprises identifications of child resources of the web page fetched by the automated browsing cluster as part of rendering the web page.
27. The system of claim 26 , wherein the page rendering data further comprises load timing information for a plurality of the child resources.
28. The system of claim 17 , wherein the automated browsing cluster operates further to:
fetch the web page a plurality of times,
identify page fetching data from each of the fetchings of the web page, and
generate the hinting feedback according to the page fetching data identified from each of the fetchings of the web page.
29. The system of claim 28 , wherein the automated browsing cluster operates further to fetch the web page the plurality of times substantially in parallel.
30. The system of claim 17 , wherein the automated browsing cluster comprises a web page fetching script having less than a full functionality of any of the web browsers, and the automated browsing cluster operates to fetch the web page using the web page fetching script.