Detection of boilerplate content
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating query recommendations. One method provides selecting one or more nodes of an object model that models a resource as a hierarchy of nodes, and determining that the selected nodes exhibit one or more predefined traits that are characteristic of boilerplate content, wherein boilerplate content comprises content that is repeated in multiple resources of a particular web site or content which is not relevant to the main content of a resource. A score associated with the selected nodes is adjusted responsive to determining that the selected nodes exhibit the predefined traits, and information is provided to a query recommendation engine, the information including textual content associated with the selected nodes, and the information identifying the adjusted score associated with the selected nodes.
1. A system comprising:
a client device; and
a computer-readable medium coupled to the client device having instructions stored thereon which, when executed by the client device, cause the client device to perform operations comprising:
receiving a resource from a server;
selecting one or more nodes of a Document Object Model (DOM) tree for the resource;
determining that the selected nodes exhibit one or more predefined traits that are characteristic of boilerplate content, wherein boilerplate content comprises content that is repeated in multiple resources of a particular web site;
adjusting a boilerplate content score associated with the selected nodes responsive to determining that the selected nodes exhibit the predefined traits that are characteristic of boilerplate content; and
providing information to a query recommendation engine, the information including textual content associated with the selected nodes, and the information identifying the adjusted boilerplate content score associated with the selected nodes.
2. The system of claim 1 , wherein the resource comprises a markup language document.
3. The system of claim 1 , wherein adjusting the boilerplate content score further comprises incrementing or decrementing the boilerplate content score based on determining that the selected nodes exhibit the predefined traits that are characteristic of boilerplate content.
4. The system of claim 1 , wherein the operations further comprise:
determining that the boilerplate content score satisfies a predetermined threshold; and
providing information to a query recommendation engine responsive to determining that the boilerplate content score satisfies the predetermined threshold, the information identifying the textual content as boilerplate content.
5. The system of claim 4 , wherein the content identified as boilerplate content comprises an anchor block.
6. The system of claim 5 , wherein determining that the selected nodes exhibit the predefined traits that are characteristic of boilerplate content further comprises determining that the selected nodes are associated with a division-type, table-type, or list-type block-level element.
7. The system of claim 5 , wherein determining that the selected nodes exhibit the predefined traits that are characteristic of boilerplate content further comprises determining that the selected nodes define a shape that includes fewer than a predetermined quantity of pixels.
8. The system of claim 5 , wherein determining that the selected nodes exhibit the predefined traits that are characteristic of boilerplate content further comprises determining that the selected nodes define a shape that exhibits a height-to-width ratio or a width-to-height ratio that is greater than a predetermined value.
9. The system of claim 5 , wherein determining that the selected nodes exhibit the predefined traits that are characteristic of boilerplate content further comprises determining that the selected nodes include at least a predetermined quantity of element objects as child nodes.
10. The system of claim 5 , wherein determining that the selected nodes exhibit the predefined traits that are characteristic of boilerplate content further comprises determining that greater than a predetermined percentage of textual content associated with the selected nodes occurs within one or more link elements.
11. The system of claim 4 , wherein the content identified as boilerplate content comprises an anchor list.
12. The system of claim 11 , wherein determining that the selected nodes exhibit the predefined traits that are characteristic of boilerplate content further comprises determining that the selected nodes include at least a predetermined quantity of anchor objects as child nodes.
13. The system of claim 11 , wherein determining that the selected nodes exhibit the predefined traits that are characteristic of boilerplate content further comprises determining that two or more anchor objects associated with the selected nodes are left aligned with each other.
14. The system of claim 4 , wherein the content identified as boilerplate content comprises a footer.
15. The system of claim 14 , wherein determining that the selected nodes exhibit the predefined traits that are characteristic of boilerplate content further comprises determining that the selected nodes define a shape that commonly occurs at a bottom of a web page.
16. The system of claim 14 , wherein determining that the selected nodes exhibit the predefined traits that are characteristic of boilerplate content further comprises determining that the selected nodes include a body element as a parent node.
17. The system of claim 1 , wherein the operations further comprise:
copying, during each of two or more time slices, different portions of the Document Object Model (DOM) tree until a complete copy of the DOM tree is obtained,
wherein selecting the nodes of the DOM tree further comprises selecting the nodes from the copy of the DOM tree, using a background thread.
18. A computer-implemented method comprising:
receiving a resource from a server;
selecting one or more nodes of a Document Object Model (DOM) tree for the resource,
determining that the selected nodes exhibit one or more predefined traits that are characteristic of boilerplate content, wherein boilerplate content comprises content that is repeated in multiple resources of a particular web site,
adjusting a boilerplate content score associated with the selected nodes responsive to determining that the selected nodes exhibit the predefined traits that are characteristic of boilerplate content, and
providing information to a query recommendation engine, the information including textual content associated with the selected nodes, and the information identifying the adjusted boilerplate content score associated with the selected nodes.
19. A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by a client device cause the client device to perform operations comprising:
receiving a resource from a server;
selecting one or more nodes of a Document Object Model (DOM) tree for the resource;
determining that the selected nodes exhibit one or more predefined traits that are characteristic of boilerplate content, wherein boilerplate content comprises content that is repeated in multiple resources of a particular web site;
adjusting a boilerplate content score associated with the selected nodes responsive to determining that the selected nodes exhibit the predefined traits that are characteristic of boilerplate content; and
providing information to a query recommendation engine, the information including textual content associated with the selected nodes, and the information identifying the adjusted boilerplate content score associated with the selected nodes.