Similar domain detection using similarity check-based ranking
Systems and methods for similar domain detection include receiving a base domain, the base domain being associated with an enterprise; receiving a domain list comprising a plurality of domains; performing a plurality of similarity checks between the base domain and each of the plurality of domains within the domain list; and generating a directory of domains comprising one or more domains determined to be associated with the enterprise based on the one or more similarity checks.
1 . A method for performing similar domain detection, the method comprising steps of:
receiving a base domain, the base domain being associated with an enterprise;
receiving a domain list comprising a plurality of domains;
performing a plurality of similarity checks between the base domain and each of the plurality of domains within the domain list, wherein performing the plurality of similarity checks comprises performing a content check that determines (i) domains that contain a full string of the base domain including a Top Level Domain (TLD) and (ii) domains that contain a domain-name portion of the base domain without the TLD, and performing one or more string-based similarity checks that compare a domain name portion of the base domain without the TLD to a domain-name portion of a respective domain without a TLD using a seq comparison that outputs a similarity ratio in a range of 0 to 1, and determining that the respective domain satisfies a corresponding similarity check when the similarity ratio exceeds a minimum ratio; and
generating a directory of domains comprising one or more domains determined to be associated with the enterprise based on the one or more similarity checks and providing the directory of domains to a computing device associated with the enterprise.
2 . The method of claim 1 , wherein the steps further comprise:
determining a proxy company used by the enterprise and extracting the domain list based thereon.
3 . The method of claim 1 , wherein the steps further comprise:
assigning one or more of the domains of the plurality of domains a similarity rank based on the plurality of similarity checks by assigning a rank selected from a plurality of ranks that are ordered from most similar to least similar based on which of the plurality of similarity checks is satisfied by a domain and based further on whether a TLD of the domain matches a TLD of the base domain and whether the TLD is a generic TLD or a non-generic TLD.
4 . The method of claim 3 , wherein the plurality of similarity checks includes a content check, a simple similarity check, and an advanced similarity check, wherein the simple similarity check comprises using a sequenceMatcher function to compute the similarity ratio between the domain-name portion of the base domain without the TLD and the domain-name portion of the respective domain without the TLD, and wherein the advanced similarity check comprises (i) computing the similarity ratio using the sequenceMatcher function with a higher minimum ratio than the simple similarity check and (ii) using a get_matching_blocks function to detect a matching sequence that is at least a threshold length and located at a beginning of both compared domain-name portions.
5 . The method of claim 3 , wherein each of the one or more domains is assigned a similarity rank based on a similarity check of the plurality of similarity checks which is satisfied.
6 . The method of claim 3 , wherein each of the one or more domains is assigned a similarity rank based on (i) whether they share a generic or unique Top Level Domain (TLD) with the base domain, (ii) whether they contain the base domain, and (iii) whether they satisfy a simple similarity check or an advanced similarity check.
7 . The method of claim 3 , wherein a domain from the domain list with a same unique Top Level Domain (TLD) as the base domain is assigned a higher rank than a domain with a same generic TLD as the base domain.
8 . The method of claim 1 , wherein generating the directory of domains is further based on a similarity rank of the one or more domains and a similarity rank threshold.
9 . The method of claim 8 , wherein the similarity rank threshold is predefined by the enterprise.
10 . The method of claim 1 , wherein the steps are performed on a per-tenant basis in a multi-tenant cloud.
11 . A non-transitory computer-readable medium comprising instructions that, when executed, cause one or more processors to perform steps of:
receiving a base domain, the base domain being associated with an enterprise;
receiving a domain list comprising a plurality of domains;
performing a plurality of similarity checks between the base domain and each of the plurality of domains within the domain list, wherein performing the plurality of similarity checks comprises performing a content check that determines (i) domains that contain a full string of the base domain including a Top Level Domain (TLD) and (ii) domains that contain a domain-name portion of the base domain without the TLD, and performing one or more string-based similarity checks that compare a domain-name portion of the base domain without the TLD to a domain name portion of a respective domain without a TLD using a sequence comparison that outputs a similarity ratio in a range of 0 to 1, and determining that the respective domain satisfies a corresponding similarity check when the similarity ratio seeds a minimum ratio; and
generating a directory of domains comprising one or more domains determined to be associated with the enterprise based on the one or more similarity checks and providing the directory of domains to a computing device associated with the enterprise.
12 . The non-transitory computer-readable medium of claim 11 , wherein the steps further comprise:
determining a proxy company used by the enterprise and extracting the domain list based thereon.
13 . The non-transitory computer-readable medium of claim 11 , wherein the steps further comprise:
assigning one or more of the domains of the plurality of domains a similarity rank based on the plurality of similarity checks by assigning a rank selected from a plurality of ranks that are ordered from most similar to least similar based on which of the plurality of similarity checks is satisfied by a domain and based further on whether a TLD of the domain matches a TLD of the base domain and whether the TLD is a generic TLD or a non-generic TLD.
14 . The non-transitory computer-readable medium of claim 13 , wherein the plurality of similarity checks includes a content check, a simple similarity check, and an advanced similarity check, wherein the simple similarity check comprises using a sequenceMatcher function to compute the similarity ratio between the domain-name portion of the base domain without the TLD and the domain-name portion of the respective domain without the TLD, and wherein the advanced similarity check comprises (i) computing the similarity ratio using the sequenceMatcher function with a higher minimum ratio than the simple similarity check and (ii) using a get_matching_blocks function to detect a matching sequence that is at least a threshold length and located at a beginning of both compared domain-name portions.
15 . The non-transitory computer-readable medium of claim 13 , wherein each of the one or more domains is assigned a similarity rank based on a similarity check of the plurality of similarity checks which is satisfied.
16 . The non-transitory computer-readable medium of claim 13 , wherein each of the one or more domains is assigned a similarity rank based on (i) whether they share a generic or unique Top Level Domain (TLD) with the base domain, (ii) whether they contain the base domain, and (iii) whether they satisfy a simple similarity check or an advanced similarity check.
17 . The non-transitory computer-readable medium of claim 13 , wherein a domain from the domain list with a same unique Top Level Domain (TLD) as the base domain is assigned a higher rank than a domain with a same generic TLD as the base domain.
18 . The non-transitory computer-readable medium of claim 11 , wherein generating the directory of domains is further based on a similarity rank of the one or more domains and a similarity rank threshold.
19 . The non-transitory computer-readable medium of claim 18 , wherein the similarity rank threshold is predefined by the enterprise.
20 . The non-transitory computer-readable medium of claim 11 , wherein the steps are performed on a per-tenant basis in a multi-tenant cloud.