Scalable derivative services
View Patent ↗An efficient method for parsing HTML pages identifies pages containing a mix of static and dynamic content. The pages are parsed to form abstract syntax trees (ASTs), which are then cached along with the pages. When a later version of a page is retrieved, it is compared against the cached version, and only those portions of the AST that contain different content are reparsed.
1. A method for efficiently identifying dynamic content of a webpage, the method comprising:
(a) accessing, by a virtual browser of a plurality of virtual browsers executing on a device intermediary to a plurality of clients and a plurality of servers a first stored data file representing a first version of a web page and a first abstract syntax tree corresponding to the first stored data file, the abstract syntax tree comprising at least one static node, the static node including stored content;
(b) identifying, by the virtual browser of the plurality of virtual browsers, non-matching dynamic content between the first stored data file and a second data file representing a second version of the web page without using a second abstract syntax tree corresponding to the second data file; and
(c) replacing, by the virtual browser, the at least one static node corresponding to the non-matching dynamic content in the first abstract syntax tree with a token that identifies the portion of the abstract syntax tree containing the non-matching dynamic content.
2. The method of claim 1 , wherein step (b) further comprises receiving, by the device, the second version of the web page from a server.
3. The method of claim 1 , wherein step (b) further comprises determining, by the virtual browser, which portions of the second version of the web page are dynamic.
4. The method of claim 3 , further comprising parsing only those portions of the second version of the web page that are dynamic.
5. The method of claim 1 , wherein step (c) further comprises obtaining, by the virtual browser, a unique token identifier for the token added to the first abstract syntax tree.
6. The method of claim 5 , further comprising maintaining, by the virtual browser, a mapping of the token to an associated subtree of the first abstract syntax tree.
7. The method of claim 6 , further comprising constructing, by the virtual browser, a dynamic subtree representing non-matching dynamic content and associating the dynamic subtree with the unique token identifier.
8. The method of claim 7 , further comprising maintaining, by the virtual browser, the association between the dynamic subtree with the unique token identifier until termination of the connection between one of the plurality of servers and the virtual browser.
9. The method of claim 1 , further comprising: (d) tracking, by the virtual browser, a ratio of static nodes to tokens for the tracked web page; (e) deleting, by the virtual browser, the first abstract syntax tree if the ratio of static nodes to tokens exceeds a threshold; and (f) building, by the virtual browser, a new abstract syntax tree.
10. The method of claim 1 , wherein step (c) further comprising storing, by the virtual browser, the first abstract syntax tree.
11. A system for efficiently identifying dynamic content of a webpage comprising:
a device intermediary to a plurality of clients and a plurality of servers;
a comparison engine of the device:
accessing a first stored data file representing a first version of a web page and first abstract syntax tree corresponding to the first stored data file, the abstract syntax tree comprising at least one static node, the static node including stored content; and
identifying non-matching dynamic content between the first stored data file and a second data file representing a second version of the web page without using a second abstract syntax tree corresponding to the second data file; and
a virtual browser of a plurality of virtual browsers executing on the device and replacing the at least one static node corresponding to the non-matching dynamic content in the first abstract syntax tree with a token that identifies the portion of the abstract syntax tree containing the non-matching dynamic content.
12. The system of claim 11 , wherein the device receives the second version of the web page from a server.
13. The system of claim 11 , wherein the virtual browser determines which portions of the second version of the web page are dynamic.
14. The system of claim 13 , wherein the virtual browser parses only those portions of the second version of the web page that are dynamic.
15. The system of claim 11 , wherein the virtual browser obtains a unique token identifier for the token added to the first abstract syntax tree.
16. The system of claim 15 , wherein the virtual browser maintains a mapping of the token to an associated subtree of the first abstract syntax tree.
17. The system of claim 16 , wherein the virtual browser constructs a dynamic subtree representing non-matching dynamic content and associating the dynamic subtree with the unique token identifier.
18. The system of claim 17 , wherein the virtual browser maintains the association between the dynamic subtree with the unique token identifier until termination of the connection between one of the plurality of servers and the virtual browser.
19. The system of claim 11 , wherein the virtual browser tracks a ratio of static nodes to tokens for the tracked web page; deletes the first abstract syntax tree if the ratio of static nodes to tokens exceeds a threshold; and builds a new abstract syntax tree.
20. The system of claim 11 , wherein the virtual browser stores the first abstract syntax tree.