Identifying a functional fragment of a document object model tree
A functional fragment of a document object model (DOM) tree is identified based on detecting at least one handler associated with at least one corresponding element of the DOM tree. In response to determining that the identified functional fragment matches a previously processed functional fragment, the identified functional fragment is excluded from further processing.
1. A method comprising:
identifying, by a system including a processor, a functional fragment of a document object model (DOM) tree, the identifying based on detecting plural handlers associated with corresponding elements of the DOM tree and generating signatures for the elements associated with the plural handlers, each handler of the plural handlers to cause execution of executable code;
determining, by the system, whether the identified functional fragment matches a previously processed functional fragment; and
in response to determining that the identified functional fragment matches the previously processed functional fragment, excluding the identified functional fragment from further processing.
2. The method of claim 1 , further comprising generating a key for the identified functional fragment, wherein the determining is based on comparing the key for the identified functional fragment to a key of the previously processed functional fragment.
3. The method of claim 2 , wherein generating the key for the identified functional fragment is based on the signatures for at least a subset of the elements.
4. The method of claim 3 , wherein generating the key for the identified functional fragment is based on applying a function to the signatures for at least the subset of the elements.
5. The method of claim 2 , further comprising comparing a structure of the identified functional fragment with a structure of the previously processed functional fragment, wherein the excluding is further in response to comparing the structures.
6. The method of claim 5 , wherein the comparing the structures comprises comparing a structure of markup language tags of the identified functional fragment with a structure of markup language tags of the previously processed functional fragment.
7. The method of claim 1 , further comprising forming clusters of the elements based on the signatures, wherein each of the clusters includes elements that have matching signatures.
8. The method of claim 1 , wherein generating the signatures comprises generating XPath strings for the elements associated with the plural handlers.
9. The method of claim 1 , wherein excluding the identified functional fragment from the further processing comprises excluding the identified functional fragment from analysis by a web crawler that analyzes dynamic web content.
10. The method of claim 1 , wherein the executable code caused to be executed by each handler of the plural handlers comprises a respective script.
11. The method of claim 1 , wherein the plural handlers are registered with the corresponding elements to be notified of respective events relating to the corresponding elements.
12. A system comprising:
at least one processor to:
identify a functional fragment of a document object model (DOM) tree, the identifying based on detecting event handlers associated with corresponding elements of the DOM tree, the event handlers to invoke respective scripts;
identify a cluster of the elements that have matching signatures;
determine whether the identified functional fragment matches a previously processed functional fragment; and
in response to determining that the identified functional fragment matches the previously processed functional fragment, exclude the identified functional fragment from further processing.
13. The system of claim 12 , wherein the further processing comprises processing of a web crawler.
14. The system of claim 12 , wherein identifying the functional fragment is based on a determination that a lowest common ancestor of the cluster of elements is not a root node of the DOM tree.
15. The system of claim 12 , wherein the identifying of the functional fragment is responsive to the cluster having a number of the elements that have the matching signatures exceeding a threshold.
16. An article comprising at least one non-transitory machine-readable storage medium storing instructions that upon execution cause a system to:
identify event handlers associated with elements of a document object model (DOM) tree, the event handlers to invoke corresponding executable code;
generate signatures for the elements;
discover a functional fragment of the DOM tree based on clustering the elements according to the generated signatures;
determine whether the discovered functional fragment matches a previously processed functional fragment; and
in response to determining that the discovered functional fragment matches the previously processed functional fragment, exclude the discovered functional fragment from processing by a web crawler.
17. The article of claim 16 , wherein the instructions upon execution cause the system to generate a key for the discovered functional fragment based on the signatures of at least a subset of the elements, and wherein determining whether the discovered functional fragment matches the previously processed function fragment uses the key.
18. The article of claim 17 , wherein generating the key for the discovered functional fragment is based on applying a function to the signatures of at least the subset of the elements.