Using hash signatures of DOM objects to identify website similarity
Embodiments are directed to using a hash signature of a rendered DOM object of a website to find similar content and behavior on other websites. Embodiments break a DOM into a large number of data portions (i.e., “shingles”), apply a hashing algorithm to the shingles, select a predetermined number of hashes from the hashed shingles according to a selection criteria to create a hash signature, and compare the hash signature to that of a reference page to determine similarity of website DOM object content. Embodiments can be used to identify phishing websites, defaced websites, spam websites, significant changes in the content of a webpage, copyright infringement, and any other suitable purposes related to the similarity between website DOM object content.
1. A method, comprising:
storing signatures of one or more portions of an original website;
retrieving a suspect website;
comparing one or more portions of the suspect website to one or more portions of the original website by:
calculating a similarity measurement between each portion of the one or more portions of the suspect website and the one or more portions of the original website;
comparing the similarity measurement to a threshold, the threshold dependent upon a type of activity being analyzed;
based on the comparing the similarity measurement to the threshold, determining that the suspect website is similar to the original website; and
based on the determination that the suspect website is similar to the original website, classifying the website with a classification associated with the threshold.
2. The method as recited in claim 1 , wherein the original website is associated with one or more classifications.
3. The method as recited in claim 1 , wherein the original website is associated with one or more classifications, wherein each classification is associated with a different similarity threshold.
4. The method as recited in claim 1 , further comprising:
storing results of the similarity measurement; and
periodically analyzing the suspect website to determine whether the suspect website is no longer associated with the classification.
5. The method as recited in claim 1 , further comprising:
storing results of the similarity measurement; and
periodically analyzing the suspect website to determine whether the suspect website has been taken down.
6. The method as recited in claim 1 , further comprising:
reporting the classification to an entity hosting a web server that is hosting the suspect website.
7. The method as recited in claim 1 , further comprising:
based on the comparing the similarity measurement to the threshold, determining that the suspect website is not similar to the original website; and
based on the determination that the suspect website is not similar to the original website, calculating a similarity measurement between each portion of the one or more portions of the suspect website and one or more portions of one or more other known websites.
8. One or more non-transitory computer-readable storage media, storing one or more sequences of instructions, which when executed by one or more processors cause performance of:
storing signatures of one or more portions of an original website;
retrieving a suspect website;
comparing one or more portions of the suspect website to one or more portions of the original website by:
calculating a similarity measurement between each portion of the one or more portions of the suspect website and the one or more portions of the original website;
comparing the similarity measurement to a threshold, the threshold dependent upon a type of activity being analyzed;
based on the comparing the similarity measurement to the threshold, determining that the suspect website is similar to the original website; and
based on the determination that the suspect website is similar to the original website, classifying the website with a classification associated with the threshold.
9. The one or more non-transitory computer-readable storage media as recited in claims 8 , wherein the original website is associated with one or more classifications.
10. The one or more non-transitory computer-readable storage media as recited in claim 8 , wherein the original website is associated with one or more classifications, wherein each classification is associated with a different similarity threshold.
11. The one or more non-transitory computer-readable storage media as recited in claim 8 , further comprising:
storing results of the similarity measurement; and
periodically analyzing the suspect website to determine whether the suspect website is no longer associated with the classification.
12. The one or more non-transitory computer-readable storage media as recited in claim 8 , further comprising:
storing results of the similarity measurement; and
periodically analyzing the suspect website to determine whether the suspect website has been taken down.
13. The one or more non-transitory computer-readable storage media as recited in claim 8 , further comprising:
reporting the classification to an entity hosting a web server that is hosting the suspect website.
14. The one or more non-transitory computer-readable storage media as recited in claim 8 , further comprising:
based on the comparing the similarity measurement to the threshold, determining that the suspect website is not similar to the original website; and
based on the determination that the suspect website is not similar to the original website, calculating a similarity measurement between each portion of the one or more portions of the suspect website and one or more portions of one or more other known websites.
15. An apparatus, comprising:
one or more processors; and
a memory storing instructions, which when executed by the one or more processors, causes the one or more processors to:
store signatures of one or more portions of an original website;
retrieve a suspect website;
compare one or more portions of the suspect website to one or more portions of the original website by:
calculating a similarity measurement between each portion of the one or more portions of the suspect website and the one or more portions of the original website;
comparing the similarity measurement to a threshold, the threshold dependent upon a type of activity being analyzed;
based on the comparing the similarity measurement to the threshold, determining that the suspect website is similar to the original website; and
based on the determination that the suspect website is similar to the original website, classifying the website with a classification associated with the threshold.
16. The apparatus as recited in claim 15 , wherein the original website is associated with one or more classifications.
17. The apparatus as recited in claim 15 , wherein the original website is associated with one or more classifications, wherein each classification is associated with a different similarity threshold.
18. The apparatus as recited in claim 15 , wherein the instructions, which when executed by the one or more processors, further causes the one or more processors to:
store results of the similarity measurement; and
periodically analyze the suspect website to determine whether the suspect website is no longer associated with the classification.
19. The apparatus as recited in claim 15 , wherein the instructions, which when executed by the one or more processors, further causes the one or more processors to:
store results of the similarity measurement; and
periodically analyze the suspect website to determine whether the suspect website has been taken down.
20. The apparatus as recited in claim 15 , wherein the instructions, which when executed by the one or more processors, further causes the one or more processors to:
report the classification to an entity hosting a web server that is hosting the suspect website.