Systems and methods for analyzing boilerplate
View Patent ↗Systems and methods for analyzing boilerplate are described. In one described system, an indexer identifies a common element in a plurality of related articles. The indexer then classifies the common element as boilerplate. For example, the indexer may identify a copyright notice appearing in a plurality of related articles. The copyright notice in these articles is considered boilerplate.
1. A computer-implemented method comprising:
identifying a common element in a plurality of articles comprising boilerplate elements and content elements;
determining that the common element is a boilerplate element of the plurality of articles by analyzing a link associated with the common element in an article of the plurality of articles, wherein analyzing the link comprises analyzing an address to which the link refers, and determining that the common element is a boilerplate element based at least in part on the link;
assigning a weight to the common element based at least in part on the determination that the common element is a boilerplate element, wherein a content element is given a higher weight than the boilerplate element;
establishing a search result set including one or more of the plurality of articles in response to a search query; and
ranking the articles included in the search result set responsive at least in part to the weight assigned to the common element.
2. The method of claim 1 , wherein the common element comprises one or more terms common to the plurality of articles and wherein determining that the common element is a boilerplate element further comprises:
analyzing a frequency of occurrence of the common terms; and
determining that the common element is a boilerplate element responsive at least in part to the frequency of occurrence of the common terms.
3. The method of claim 2 , wherein the common element is determined to be a boilerplate element responsive at least in part to the terms having a low inverse document frequency measure.
4. The method of claim 1 , wherein determining that the common element is a boilerplate element further comprises:
determining whether a term within the common element is within a predetermined list of terms; and
determining that the common element is a boilerplate element responsive at least in part to the term being within the predetermined list of terms.
5. A non-transitory computer-readable storage medium storing executable program code comprising code for:
identifying a common element in a plurality of articles comprising boilerplate elements and content elements;
determining that the common element is a boilerplate element of the plurality of articles by analyzing a link associated with the common element in an article of the plurality of articles, wherein analyzing the link comprises analyzing an address to which the link refers, and determining that the common element is a boilerplate element based at least in part on the link;
assigning a weight to the common element based at least in part on the determination that the common element is a boilerplate element, wherein a content element is given a higher weight than the boilerplate element;
establishing a search result set including one or more of the plurality of articles in response to a search query; and
ranking the articles included in the search result set responsive at least in part to the weight assigned to the common element.
6. The computer-readable storage medium of claim 5 , wherein the common element comprises one or more terms common to the plurality of articles and determining that the common element is a boilerplate element further comprises:
analyzing a frequency of occurrence of the common terms; and
determining that the common element is a boilerplate element responsive at least in part to the frequency of occurrence of the common terms.
7. The computer-readable storage medium of claim 6 , wherein the element is determined to be a boilerplate element responsive at least in part to the terms having a low inverse document frequency measure.
8. The computer-readable storage medium of claim 5 , wherein determining that the common element is a boilerplate element further comprises:
determining whether a term within the common element is within a predetermined list of terms; and
determining that the common element is a boilerplate element responsive at least in part to the term being within the predetermined list of terms.
9. A computer system comprising:
a non-transitory computer-readable storage medium storing executable program code comprising code for:
identifying a common element in a plurality of articles comprising boilerplate elements and content elements;
determining that the common element is a boilerplate element of the plurality of articles at least by analyzing a link associated with the common element in an article of the plurality of articles, wherein analyzing the link comprises analyzing an address to which the link refers, and determining that the common element is a boilerplate element based at least in part on the link;
assigning a weight to the common element based at least in part on the determination that the common element is a boilerplate element, wherein a content element is given a higher weight than the boilerplate element;
establishing a search result set including one or more of the plurality of articles in response to a search query; and
ranking the articles included in the search result set responsive at least in part to the weight assigned to the common element; and
a processor for executing the program code.
10. The computer system of claim 9 , wherein the common element comprises one or more terms common to the plurality of articles and determining that the common element is a boilerplate element further comprises:
analyzing a frequency of occurrence of the common terms; and
determining that the common element is a boilerplate element responsive at least in part to the frequency of occurrence of the common terms.
11. The computer system of claim 10 , wherein the common element is determined to be a boilerplate element responsive at least in part to the terms having a low inverse document frequency measure.
12. The computer system of claim 9 , wherein determining that the common element is a boilerplate element further comprises:
determining whether a term within the common element is within a predetermined list of terms; and
determining that the common element is a boilerplate element responsive at least in part to the term being within the predetermined list of terms.