Spam flood detection methodologies
A computer-implemented method and system are provided in which characteristics of a website are analyzed to determine whether the website represents a potential source of spam content. The analysis can include generating a characterizing signature of a webpage containing a content item, and obtaining an occurrence count for the generated characterizing signature. The characterizing signature is derived from formatting data of the webpage. When the obtained occurrence count is greater than a threshold count, the content item can be identified as spam content, and flagged as spam content.
1. A computer-implemented method comprising:
analyzing characteristics of a content item of a webpage to determine whether the content item represents a potential source of spam content, wherein the analyzing comprises:
selecting, in accordance with predefined selection criteria, a plurality of hypertext markup language (HTML) tags of the webpage;
creating an ordered sequence of the selected plurality of HTML tags; and
applying a hash function to the ordered sequence of the selected plurality of HTML tags to map the selected plurality of HTML tags to a characterizing signature of the webpage;
updating an occurrence count for the characterizing signature in response to the characterizing signature being generated; and
when the occurrence count is greater than a threshold count, identifying the content item as spam content; and
flagging the content item of the webpage as spam content.
2. The computer-implemented method of claim 1 , further comprising:
filtering the spam content from stored webcrawler data to obtain filtered webcrawler data; and
outputting the filtered webcrawler data for presentation to a user.
3. The computer-implemented method of claim 1 , wherein the analyzing further comprises:
when the occurrence count is less than or equal to the threshold count, identifying the content item as non-spam content.
4. The computer-implemented method of claim 1 , wherein the updating comprises:
incrementing the occurrence count each time the characterizing signature of the webpage is generated.
5. A computing system comprising a processor and a memory having computer-executable instructions stored thereon that, when executed by the processor, cause the computing system to:
analyze characteristics of a content item of a webpage to determine whether the content item represents a potential source of spam content by causing the computing system to:
select, in accordance with predefined selection criteria, a plurality of hypertext markup language (HTML) tags of the webpage;
create an ordered sequence of the selected plurality of HTML tags;
map the ordered sequence of the selected plurality of HTML tags to a characterizing signature of a webpage by applying a hash function to the ordered sequence of the selected plurality of HTML tags to obtain a hash value that corresponds to the characterizing signature;
update an occurrence count for the characterizing signature in response to the characterizing signature being generated; and
when the occurrence count is greater than a threshold count, identify the content item as spam content; and
flag the content item of the webpage as spam content.
6. The computing system of claim 5 , wherein the computer-executable instructions cause the computing system to:
filter the spam content from stored webcrawler data to obtain filtered webcrawler data; and
output the filtered webcrawler data for presentation to a user.
7. The computing system of claim 5 , wherein the computer-executable instructions cause the computing system to:
identify, when the occurrence count is less than or equal to the threshold count, the content item as non-spam content.
8. The computing system of claim 5 , wherein the computer-executable instructions cause the computing system to:
increment the occurrence count each time the characterizing signature of the webpage is generated.
9. A tangible and non-transitory computer readable medium having computer-executable instructions stored thereon that, when executed by a processor, perform a method comprising:
analyzing characteristics of a content item of a webpage to determine whether the content item represents a potential source of spam content, wherein the analyzing comprises:
selecting, in accordance with predefined selection criteria, a plurality of hypertext markup language (HTML) tags of a webpage;
creating an ordered sequence of the selected plurality of HTML tags; and
applying a hash function to the ordered sequence of the selected plurality of HTML tags to map the selected plurality of HTML tags to a characterizing signature of the webpage;
updating an occurrence count for the characterizing signature in response to the characterizing signature being generated; and
when the occurrence count is greater than a threshold count, identifying the content item as spam content; and
flagging the content item of the webpage as spam content.
10. The tangible and non-transitory computer readable medium of claim 9 , the method further comprising:
filtering the spam content from stored webcrawler data to obtain filtered webcrawler data; and
outputting the filtered webcrawler data for presentation to a user.
11. The tangible and non-transitory computer readable medium of claim 9 , wherein the analyzing further comprises:
when the occurrence count is less than or equal to the threshold count, identifying the content item as non-spam content.
12. The tangible and non-transitory computer readable medium of claim 9 , wherein updating further comprises:
incrementing the occurrence count each time the characterizing signature of the webpage is generated.