Natural language processing systems and methods for automatic reduction of false positives in domain discovery
A rules engine is adapted for analyzing each match produced by a domain discovery system as matching a seed domain. Utilizing a natural language processing (NLP) library, the rules engine determines segments from the match, assigns a lexical category to each segment based on the context in how a seed domain string is used, and compares the lexical category of the segment that is closest to the seed domain string with a lexical category of the seed domain string. Based on the comparing, the rules engine determines whether the match is relevant to the seed domain and, if not, the match produced by the domain discovery system is identified as a false positive and automatically removed from a set of matches produced by the domain discovery system for the seed domain.
1. A method for automatically reducing false positives in a domain discovery process, the method comprising:
analyzing, by a rules engine, a match produced by a domain discovery system, the match comprising a domain name determined by the domain discovery system as matching a seed domain, the seed domain having a seed domain string;
extracting, by the rules engine utilizing a natural language processing (NLP) library, a sequence of segments from the match, wherein the extracting excludes a top-level domain (TLD) of the domain name;
assigning, by the rules engine utilizing the NLP library, a lexical category to each segment of the sequence of segments;
determining, by the rules engine through fuzzy string matching, a segment of the sequence of segments that is closest to the seed domain string;
comparing, by the rules engine, the lexical category of the segment of the sequence of segments that is closest to the seed domain string with a lexical category of the seed domain string;
determining, by the rules engine based on the comparing, whether the match is relevant to the seed domain;
in response to a determination by the rules engine that the match is not relevant to the seed domain, identifying, by the rules engine, the match produced by the domain discovery system as a false positive; and
automatically removing the false positive from a set of matches produced by the domain discovery system for the seed domain.
2. The method according to claim 1 , further comprising:
performing the analyzing, the extracting, the assigning, the determining, and the comparing for each match of the set of matches;
clustering non-relevant matches in the set of matches, wherein the non-relevant matches are determined by the rules engine as not being relevant to the seed domain; and
removing the non-relevant matches thus clustered from the set of matches.
3. The method according to claim 1 , further comprising:
prior to the extracting, excluding the TLD from the domain name of the match determined by the domain discovery system as matching the seed domain.
4. The method according to claim 1 , wherein the automatically removing the false positive from the set of matches comprises dissociating the false positive and the seed domain.
5. The method according to claim 4 , further comprising:
dissociating false positives identified by the rules engine from being associated with the seed domain, the dissociating producing a final result set of relevant matches for the seed domain.
6. The method according to claim 5 , further comprising:
providing the final result set of relevant matches for the seed domain to a user interface or as input to a computing facility downstream from the rules engine.
7. The method according to claim 1 , wherein each match of the set of matches produced by the domain discovery system for the seed domain contains a segment that is identical to the seed domain string.
8. A system for automatically reducing false positives in a domain discovery process, the system comprising:
a processor;
a non-transitory computer-readable medium; and
stored instructions translatable by the processor for:
analyzing a match produced by the domain discovery process, the match comprising a domain name that matches a seed domain, the seed domain having a seed domain string;
extracting, utilizing a natural language processing (NLP) library, a sequence of segments from the match, wherein the extracting excludes a top-level domain (TLD) of the domain name;
assigning, utilizing the NLP library, a lexical category to each segment of the sequence of segments;
determining, through fuzzy string matching, a segment of the sequence of segments that is closest to the seed domain string;
comparing the lexical category of the segment of the sequence of segments that is closest to the seed domain string with a lexical category of the seed domain string;
determining, based on the comparing, whether the match is relevant to the seed domain;
in response to a determination that the match is not relevant to the seed domain, identifying the match produced by the domain discovery process as a false positive; and
automatically removing the false positive from a set of matches produced by the domain discovery process for the seed domain.
9. The system of claim 8 , wherein the stored instructions translatable by the processor for:
performing the analyzing, the extracting, the assigning, the determining, and the comparing for each match of the set of matches;
clustering non-relevant matches in the set of matches, wherein the non-relevant matches are determined by the rules engine as not being relevant to the seed domain; and
removing the non-relevant matches thus clustered from the set of matches.
10. The system of claim 8 , wherein the stored instructions translatable by the processor for:
prior to the extracting, excluding the TLD from the domain name of the match determined by the domain discovery system as matching the seed domain.
11. The system of claim 8 , wherein the automatically removing the false positive from the set of matches comprises dissociating the false positive and the seed domain.
12. The system of claim 11 , wherein the stored instructions translatable by the processor for:
performing the dissociating for each of the set of matches determined as not being relevant to the seed domain to thereby produce a final result set of relevant matches for the seed domain.
13. The system of claim 12 , wherein the stored instructions translatable by the processor for:
providing the final result set of relevant matches for the seed domain to a user interface or as input to a computing facility.
14. The system of claim 8 , wherein each match of the set of matches produced by the domain discovery process for the seed domain contains a segment that is identical to the seed domain string.
15. A computer program product comprising a non-transitory computer-readable medium storing instructions translatable by a processor to perform:
analyzing a match produced by a domain discovery process, the match comprising a domain name that matches a seed domain, the seed domain having a seed domain string;
extracting, utilizing a natural language processing (NLP) library, a sequence of segments from the match, wherein the extracting excludes a top-level domain (TLD) of the domain name;
assigning, utilizing the NLP library, a lexical category to each segment of the sequence of segments;
determining, through fuzzy string matching, a segment of the sequence of segments that is closest to the seed domain string;
comparing the lexical category of the segment of the sequence of segments that is closest to the seed domain string with a lexical category of the seed domain string;
determining, based on the comparing, whether the match is relevant to the seed domain;
in response to a determination that the match is not relevant to the seed domain, identifying the match produced by the domain discovery process as a false positive; and
automatically removing the false positive from a set of matches produced by the domain discovery process for the seed domain.
16. The computer program product of claim 15 , wherein the instructions translatable by the processor for:
performing the analyzing, the extracting, the assigning, the determining, and the comparing for each match of the set of matches;
clustering non-relevant matches in the set of matches, wherein the non-relevant matches are determined by the rules engine as not being relevant to the seed domain; and
removing the non-relevant matches thus clustered from the set of matches.
17. The computer program product of claim 15 , wherein the stored instructions translatable by the processor for:
prior to the extracting, excluding the TLD from the domain name of the match determined by the domain discovery system as matching the seed domain.
18. The computer program product of claim 15 , wherein the automatically removing the false positive from the set of matches comprises dissociating the false positive and the seed domain.
19. The computer program product of claim 15 , wherein the instructions translatable by the processor for:
performing the dissociating for each of the set of matches determined as not being relevant to the seed domain to thereby produce a final result set of relevant matches for the seed domain; and
providing the final result set of relevant matches for the seed domain to a user interface or as input to a computing facility.
20. The computer program product of claim 15 , wherein each match of the set of matches produced by the domain discovery process for the seed domain contains a segment that is identical to the seed domain string.