Using hash signatures of DOM objects to identify website similarity
Embodiments are directed to using a hash signature of a rendered DOM object of a website to find similar content and behavior on other websites. Embodiments break a DOM into a large number of data portions (i.e., “shingles”), apply a hashing algorithm to the shingles, select a predetermined number of hashes from the hashed shingles according to a selection criteria to create a hash signature, and compare the hash signature to that of a reference page to determine similarity of website DOM object content. Embodiments can be used to identify phishing websites, defaced websites, spam websites, significant changes in the content of a webpage, copyright infringement, and any other suitable purposes related to the similarity between website DOM object content.
1. A method comprising, at a computer system:
rendering a document object model (DOM) object of a website using website information;
generating, by a hardware processor of the computer system, a hash signature of the DOM object of the website by:
applying a number of hashing functions to each of a plurality of data portions separated from content within the DOM object of the website; and
generating a hash signature of the DOM object of the website by selecting a number of hashed data portions of the plurality of hashed data portions;
obtaining a known hash signature of a DOM object associated with each of a plurality of known websites associated with a classification;
comparing the hash signature of the DOM object of the website to a known hash signature of the DOM object associated with each of the plurality of known websites, wherein comparing the hash signature of the DOM object of the website to the known hash signature of the DOM object associated with each of the plurality of known websites includes comparing each of the plurality of hashed data portions to a plurality of known hashed data portions of the known hash signature;
based on determining that the hash signature of the DOM object of the website does not match the known hash signature of the DOM object associated with each of the plurality of known websites:
for each of the plurality of known websites associated with the classification:
computing a similarity measurement between the hash signature of the DOM object of the website and the known hash signature of the DOM object associated with the known websites associated with the classification;
determining whether a threshold associated with the classification is satisfied based on the similarity measurement computed between the hash signature of the DOM object of the website and the known hash signature of the DOM object for each of the plurality of known websites; and
determining whether to classify the website as confirmed for the classification based on determining whether the threshold associated with the classification is satisfied.
2. The method of claim 1 , wherein the website is classified as confirmed for the classification based on determining that the threshold associated with the classification is satisfied, and wherein the website is classified as dismissed for the classification based on determining that the threshold associated with the classification is not satisfied.
3. The method of claim 1 , further comprising:
storing the hash signature of the DOM object for the website in association with the classification based on determining that the website is classified as confirmed for the classification.
4. The method of claim 1 , wherein the similarity measurement includes a magnitude of each of the plurality of hashed data portions that are shared between the hash signature of the DOM object of the website and the known hash signature of the DOM object associated with the known website divided by a magnitude of the plurality of hashed data portions that are shared.
5. The method of claim 1 , further comprising:
determining a predetermined number of most similar known hash signatures based on the similarity measurement computed between the hash signature of the DOM object of the website and the known hash signature of the DOM object associated with each of the plurality of known websites; and
computing a similarity weight for the predetermined number of the most similar known hash signatures; and
wherein determining whether the threshold associated with the classification is satisfied based on the similarity measurement computed between the hash signature of the DOM object of the website and the known hash signature of the DOM object associated with each of the plurality of known web sites includes determining whether the similarity weight satisfies the threshold associated with the classification.
6. The method of claim 5 , wherein computing a similarity weight for the predetermined number of the most similar known hash signatures includes:
weighting a similarity of the predetermined number of the most similar known hash signatures according to a distribution of similarity measurements of the predetermined number of the most similar known hash signatures; and
calculating the similarity weight for the most similar known hash signatures by dividing a sum of similarity weights of a number of the most similar known hash signatures that are classified as confirmed by a sum of similarity weights of a total number of the most similar known hash signatures.
7. The method of claim 1 , wherein the number of hashing functions applied to each of a plurality of data portions is a predetermined number of hashing functions that are generated using a common seed value, and wherein applying the number of hashing functions results in a predetermined number of values for each of the plurality of data portions.
8. The method of claim 1 , wherein each of the plurality of data portions separated from the content within the DOM object of the website have a fixed length.
9. A system comprising:
one or more hardware processors; and
a memory device accessible to the one or more hardware processors, wherein the memory device stores one or more instructions that, upon execution by the one or more hardware processors, causes the one or more hardware processors to:
render a document object model (DOM) object of a website using website information;
generate a hash signature of the DOM object of the website by:
applying a number of hashing functions to each of a plurality of data portions separated from content within the DOM object; and
generating a hash signature of the DOM object of the website by selecting a number of hashed data portions of the plurality of hashed data portions;
obtain a known hash signature of a DOM object associated with each of a plurality of known websites associated with a classification;
compare the hash signature of the DOM object of the website to a known hash signature of the DOM object associated with each of the plurality of known websites, wherein comparing the hash signature of the DOM object of the website to the known hash signature of the DOM object associated with each of the plurality of known web sites includes comparing each of the plurality of hashed data portions to a plurality of known hashed data portions of the known hash signature;
based on determining that the hash signature of the DOM object of the website does not match the known hash signature of the DOM object associated with each of the plurality of known websites:
for each of the plurality of known websites associated with the classification:
compute a similarity measurement between the hash signature of the DOM object of the website and the known hash signature of the DOM object associated with the known websites associated with the classification;
determine whether a threshold associated with the classification is satisfied based on the similarity measurement computed between the hash signature of the DOM object of the web site and the known hash signature of the DOM object for each of the plurality of known websites; and
determine whether to classify the website as confirmed for the classification based on determining whether the threshold associated with the classification is satisfied.
10. The system of claim 9 , wherein the website is classified as confirmed for the classification based on determining that the threshold associated with the classification is satisfied, and wherein the website is classified as dismissed for the classification based on determining that the threshold associated with the classification is not satisfied.
11. The system of claim 9 , wherein the similarity measurement includes a magnitude of each of the plurality of hashed data portions that are shared between the hash signature of the DOM object of the website and the known hash signature of the DOM object associated with the known website divided by a magnitude of the plurality of hashed data portions that are shared.
12. The system of claim 9 , wherein the one or more instructions, upon execution by the one or more hardware processors, further causes the one or more hardware processors to:
determine a predetermined number of most similar known hash signatures based on the similarity measurement computed between the hash signature of the DOM object of the website and the known hash signature of the DOM object associated with each of the plurality of known websites; and
compute a similarity weight for the predetermined number of the most similar known hash signatures; and
wherein determining whether the threshold associated with the classification is satisfied based on the similarity measurement computed between the hash signature of the DOM object of the website and the known hash signature of the DOM object associated with each of the plurality of known websites includes determining whether the similarity weight satisfies the threshold associated with the classification.
13. The system of claim 12 , wherein computing a similarity weight for the predetermined number of the most similar known hash signatures includes:
weighting a similarity of the predetermined number of the most similar known hash signatures according to a distribution of similarity measurements of the predetermined number of the most similar known hash signatures; and
calculating the similarity weight for the most similar known hash signatures by dividing a sum of similarity weights of a number of the most similar known hash signatures that are classified as confirmed by a sum of similarity weights of a total number of the most similar known hash signatures.
14. The system of claim 9 , wherein the number of hashing functions applied to each of a plurality of data portions is a predetermined number of hashing functions that are generated using a common seed value, and wherein applying the number of hashing functions results in a predetermined number of values for each of the plurality of data portions.
15. The system of claim 9 , wherein the classification is a first classification as a phishing website, a second classification as copyrighted content, or a third classification as unauthorized content.
16. A non-transitory computer-readable storage medium storing one or more instructions that, upon execution by one or more processors, causes the one or more processors to:
render a document object model (DOM) object of a website using website information;
generate a hash signature of the DOM object of the website by:
applying a number of hashing functions to each of a plurality of data portions separated from content within the DOM object of the website; and
generating a hash signature of the DOM object of the website by selecting a number of hashed data portions of the plurality of hashed data portions;
obtain a known hash signature of a DOM object associated with each of a plurality of known websites associated with a classification;
compare the hash signature of the DOM object of the website to a known hash signature of the DOM object associated with each of the plurality of known websites, wherein comparing the hash signature of the DOM object of the website to the known hash signature of the DOM object associated with each of the plurality of known websites includes comparing each of the plurality of hashed data portions to a plurality of known hashed data portions of the known hash signature;
based on determining that the hash signature of the DOM object of the website does not match the known hash signature of the DOM object associated with each of the plurality of known websites:
for each of the plurality of known websites associated with the classification:
compute a similarity measurement between the hash signature of the DOM object of the website and the known hash signature of the DOM object associated with the known websites associated with the classification;
determine whether a threshold associated with the classification is satisfied based on the similarity measurement computed between the hash signature of the DOM object of the website and the known hash signature of the DOM object for each of the plurality of known websites; and
determine whether to classify the website as confirmed for the classification based on determining whether the threshold associated with the classification is satisfied.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the website is classified as confirmed for the classification based on determining that the threshold associated with the classification is satisfied, and wherein the website is classified as dismissed for the classification based on determining that the threshold associated with the classification is not satisfied.
18. The non-transitory computer-readable storage medium of claim 16 , wherein the one or more instructions, upon execution by the one or more processors, further causes the one or more processors to:
determine a predetermined number of most similar known hash signatures based on the similarity measurement computed between the hash signature of the DOM object of the website and the known hash signature of the DOM object associated with each of the plurality of known websites; and
compute a similarity weight for the predetermined number of the most similar known hash signatures; and
wherein determining whether the threshold associated with the classification is satisfied based on the similarity measurement computed between the hash signature of the DOM object of the website and the known hash signature of the DOM object associated with each of the plurality of known web sites includes determining whether the similarity weight satisfies the threshold associated with the classification.
19. The non-transitory computer-readable storage medium of claim 16 , wherein the number of hashing functions applied to each of a plurality of data portions is a predetermined number of hashing functions that are generated using a common seed value, and wherein applying the number of hashing functions results in a predetermined number of values for each of the plurality of data portions.
20. The non-transitory computer-readable storage medium of claim 16 , wherein the classification is a first classification as a phishing website, a second classification as copyrighted content, or a third classification as unauthorized content.