Using hash signatures of DOM objects to identify website similarity
Embodiments are directed to using a hash signature of a rendered DOM object of a website to find similar content and behavior on other websites. Embodiments break a DOM into a large number of data portions (i.e., “shingles”), apply a hashing algorithm to the shingles, select a predetermined number of hashes from the hashed shingles according to a selection criteria to create a hash signature, and compare the hash signature to that of a reference page to determine similarity of website DOM object content. Embodiments can be used to identify phishing websites, defaced websites, spam websites, significant changes in the content of a webpage, copyright infringement, and any other suitable purposes related to the similarity between website DOM object content.
1. A method comprising, at a computer system:
rendering a document object model (DOM) object for a document;
generating, by a hardware processor of the computer system, a hash signature of the DOM object by:
separating content of the DOM object into a plurality of data portions;
applying multiple hashing functions to each of the plurality of data portions, wherein applying the multiple hashing functions results in multiple values for each of the plurality of data portions, wherein each of the multiple hashing functions is different; and
generating a hash signature of the DOM object of the document based on selecting a subset of the multiple values;
comparing the hash signature of the DOM object to a known hash signature of a DOM object associated with a known document having a classification;
calculating a similarity measurement between the hash signature of the DOM object and the known hash signature of the DOM object associated with the known document;
comparing the similarity measurement to a threshold; and
determining that the document has the classification based on the similarity measurement satisfying the threshold.
2. The method of claim 1 , wherein the document is an email document.
3. The method of claim 1 , wherein the document is a web-based document.
4. The method of claim 1 , further comprising:
receiving information from a server computer corresponding to a website, wherein the DOM object of the document is rendered using the information received from the server computer.
5. The method of claim 1 , wherein each of a plurality of data portions has a fixed length, and wherein a hashing function of the multiple hashing functions is selected based on a seed value.
6. The method of claim 1 , wherein the classification of the known document is one of a trademark classification for trademark content or a copyright classification for copyrighted content.
7. The method of claim 1 , wherein the classification of the known document is a website revision classification.
8. The method of claim 1 , wherein the classification of the known document is a content classification for unauthorized content.
9. The method of claim 1 , further comprising:
determining that the document does not have the classification based on the similarity measurement not satisfying the threshold.
10. The method of claim 1 , wherein the threshold is based on the classification of the known document.
11. The method of claim 1 , wherein a number of hashing functions used is based on the threshold for the classification of the known document.
12. The method of claim 1 , wherein the known hash signature is a first known hash signature of the DOM object associated with the known document, wherein the known document is a first known document, wherein the DOM object associated with the first known document is a first DOM object, wherein the classification is a first classification, wherein the similarity measurement is a first similarity measurement, wherein the threshold is a first threshold based on the first classification, and wherein the method further comprises:
comparing the hash signature of the DOM object to a second known hash signature of a second DOM object associated with a second known document having a second classification;
calculating a second similarity measurement between the hash signature of the DOM object and the second known hash signature of the second DOM object associated with the second known document;
comparing the second similarity measurement to a second threshold based on the second classification; and
determining whether the document has the second classification based on whether the second similarity measurement satisfies the second threshold.
13. The method of claim 12 , wherein the first classification is different from the second classification, and wherein the first threshold is different from the second threshold.
14. The method of claim 12 , wherein the second known hash signature is different from the first known hash signature.
15. The method of claim 12 , wherein the document does not have the second classification based on the second similarity measurement not satisfying the second threshold.
16. The method of claim 15 , wherein the first classification is a phishing classification, and wherein the second classification is one of a trademark classification for trademark content or a copyright classification for copyrighted content.
17. A computer system comprising:
one or more processors; and
a memory accessible to the one or more processors, wherein the memory stores one or more instructions that, upon execution by the one or more processors, causes the one or more processors to:
render a document object model (DOM) object for a document;
generate a hash signature of the DOM object by:
separating content of the DOM object into a plurality of data portions;
applying multiple hashing functions to each of the plurality of data portions, wherein applying the multiple hashing functions results in multiple values for each of the plurality of data portions, wherein each of the multiple hashing functions is different; and
generating a hash signature of the DOM object of the document based on selecting a subset of the multiple values;
compare the hash signature of the DOM object to a known hash signature of a DOM object associated with a known document having a classification;
calculate a similarity measurement between the hash signature of the DOM object and the known hash signature of the DOM object associated with the known document;
compare the similarity measurement to a threshold; and
determine that the document has the classification based on the similarity measurement satisfying the threshold.
18. The computer system of claim 17 , wherein the document is an email document, wherein the threshold is based on the classification, and wherein the classification of the known document is one of a trademark classification for trademark content or a copyright classification for copyrighted content.
19. An apparatus comprising: a memory;
one or more hardware processors configured to:
render a document object model (DOM) object for a document;
generate a hash signature of the DOM object by:
separating content of the DOM object into a plurality of data portions;
applying multiple hashing functions to each of the plurality of data portions, wherein applying the multiple hashing functions results in multiple values for each of the plurality of data portions, wherein each of the multiple hashing functions is different; and
generating a hash signature of the DOM object of the document based on selecting a subset of the multiple values;
compare the hash signature of the DOM object to a known hash signature of a DOM object associated with a known document having a classification;
calculate a similarity measurement between the hash signature of the DOM object and the known hash signature of the DOM object associated with the known document;
compare the similarity measurement to a threshold; and
determine that the document has the classification based on the similarity measurement satisfying the threshold.
20. The apparatus of claim 19 , wherein the known hash signature is a first known hash signature of the DOM object associated with the known document, wherein the known document is a first known document, wherein the DOM object associated with the first known document is a first DOM object, wherein the classification is a first classification, wherein the similarity measurement is a first similarity measurement, wherein the threshold is a first threshold based on the first classification, and wherein the one or more hardware processors is further configured to:
compare the hash signature of the DOM object to a second known hash signature of a second DOM object associated with a second known document having a second classification, wherein the first classification is different from the second classification;
calculate a second similarity measurement between the hash signature of the DOM object and the second known hash signature of the second DOM object associated with the second known document;
compare the second similarity measurement to a second threshold based on the second classification, wherein the first threshold is different from the second threshold; and
determine whether the document has the second classification based on whether the second similarity measurement satisfies the second threshold.