Linguistic nonsense detection for undesirable message classification
View Patent ↗Nonsense words are removed from incoming emails and visually similar (look-alike) characters are replaced with the actual, corresponding characters, so that the emails can be more accurately analyzed to see if they are spam. More specifically, an incoming email stream is filtered, and the emails are normalized to enable more accurate spam detection. In some embodiments, the normalization comprises the removal of nonsense words and/or the replacement of look-alike characters according to a set of rules. In other embodiments, more and/or different normalization techniques are utilized. In some embodiments, the language in which an email is written is identified in order to aid in the normalization. Once incoming emails are normalized, they are then analyzed to detect spam or other forms of undesirable email, such as phishing emails.
1. A computer implemented method for identifying undesirable electronic messages by a computer, the method comprising the steps of:
identifying, by the computer, incoming electronic messages;
normalizing, by the computer, identified electronic messages according to a plurality of rules for distinguishing non-legitimate words from legitimate words, said normalizing further comprising identifying non-legitimate words obfuscating electronic messages according to the plurality of rules, and deleting the identified non-legitimate words from the electronic messages;
wherein said plurality of rules for distinguishing non-legitimate words from legitimate words comprises at least three rules from a group of rules consisting of:
a rule specifying a maximum number of consecutive vowels in a legitimate word; a rule specifying a maximum number of consecutive consonants in a legitimate word; a rule specifying a maximum number of consecutive uses of any single character in a legitimate word; a rule specifying a maximum number of transitions between upper case letters and lower case letters in a legitimate word; a rule specifying a maximum length of a legitimate word containing numbers without punctuation; a rule specifying a maximum length of a legitimate word containing upper case letters, lower case letters and numbers; a rule specifying a maximum length of a legitimate word containing upper case letters, lower case letters, numbers and punctuation; a rule specifying a minimum number of vowels in a legitimate word; a rule specifying a minimum number of consonants in a legitimate word; a rule specifying a minimum ratio of vowels to consonants in a legitimate word; and a rule specifying a maximum ratio of vowels to consonants in a legitimate word; and
analyzing, by the computer, normalized electronic message to identify undesirable electronic messages.
2. The method of claim 1 further comprising:
identifying, by the computer, a language in which an identified electronic message is composed, prior to normalizing the identified electronic message according to the set of rules; and
normalizing, by the computer, the identified electronic messages according to a set of rules taking into account the identified language.
3. The method of claim 1 wherein normalizing identified electronic messages according to a set of rules further comprises:
identifying, by the computer, look-alike characters in electronic messages; and
replacing, by the computer, identified look-alike characters with corresponding primary characters in the electronic messages.
4. The method of claim 1 wherein analyzing normalized electronic message to identify undesirable electronic messages further comprises:
performing, by the computer, a statistical analysis of normalized electronic message content in order to identify statistical patterns associated with undesirable electronic messages.
5. The method of claim 1 wherein analyzing normalized electronic message to identify undesirable electronic messages further comprises:
performing, by the computer, a hash based analysis of normalized electronic message content to detect received duplicate copies of electronic messages.
6. The method of claim 1 wherein analyzing normalized electronic message to identify undesirable electronic messages further comprises:
keeping, by the computer, a count of detected items to be normalized in an electronic message; and
using, by the computer, data concerning a relationship between the count of detected items and a dynamic threshold value as evidence in the identification of an electronic message as undesirable.
7. The method of claim 1 further comprising determining, by the computer, that at least one received electronic message comprises a type of undesirable electronic message from a group of undesirable electronic message types consisting of:
a spam email; and
a phishing email.
8. At least one non-transitory medium storing a computer program product for identifying undesirable electronic messages, the computer program product comprising:
program code for identifying incoming electronic messages;
program code for normalizing identified electronic messages according to a plurality rules for distinguishing non-legitimate words from legitimate words, said normalizing further comprising identifying non-legitimate words obfuscating electronic messages according to the plurality of rules and deleting the identified non-legitimate words from the electronic messages;
wherein said plurality of rules for distinguishing non-legitimate words from legitimate words comprises at least three rules from a group of rules consisting of:
a rule specifying a maximum number of consecutive vowels in a legitimate word;
a rule specifying a maximum number of consecutive consonants in a legitimate word;
a rule specifying a maximum number of consecutive uses of any single character in a legitimate word;
a rule specifying a maximum number of transitions between upper case letters and lower case letters in a legitimate word;
a rule specifying a maximum length of a legitimate word containing numbers without punctuation; a rule specifying a maximum length of a legitimate word containing upper case letters, lower case letters and numbers;
a rule specifying a maximum length of a legitimate word containing upper case letters, lower case letters, numbers and punctuation;
a rule specifying a minimum number of vowels in a legitimate word;
a rule specifying a minimum number of consonants in a legitimate word;
a rule specifying a minimum ratio of vowels to consonants in a legitimate word;
and a rule specifying a maximum ratio of vowels to consonants in a legitimate word; and
program code for analyzing normalized electronic message to identify undesirable electronic messages.
9. The computer program product of claim 8 further comprising:
program code for identifying a language in which an identified electronic message is composed, prior to normalizing the identified electronic message according to the set of rules; and
program code for normalizing the identified electronic messages according to a set of rules taking into account the identified language.
10. The computer program product of claim 8 wherein the program code for normalizing identified electronic messages according to a set of rules further comprises:
program code for identifying look-alike characters in electronic messages; and
program code for replacing identified look-alike characters with corresponding primary characters in the electronic messages.
11. The computer program product of claim 8 wherein the program code for analyzing normalized electronic message to identify undesirable electronic messages further comprises:
program code for performing a statistical analysis of normalized electronic message content in order to identify statistical patterns associated with undesirable electronic messages.
12. The computer program product of claim 8 wherein the program code for analyzing normalized electronic message to identify undesirable electronic messages further comprises:
program code for performing a hash based analysis of normalized electronic message content to detect received duplicate copies of electronic messages.
13. The computer program product of claim 8 wherein the program code for analyzing normalized electronic message to identify undesirable electronic messages further comprises:
program code for keeping a count of detected items to be normalized in an electronic message; and
program code for using data concerning a relationship between the count of detected items and a dynamic threshold value as evidence in the identification of an electronic message as undesirable.
14. The computer program product of claim 8 further comprising program code for determining that at least one received electronic message comprises a type of undesirable electronic message from a group of undesirable electronic message types consisting of:
a spam email; and
a phishing email.
15. A computer system for identifying undesirable electronic messages, the computer system comprising:
means for identifying incoming electronic messages;
means for normalizing identified electronic messages according to a plurality of rules for distinguishing non-legitimate words from legitimate words, said normalizing further comprising identifying non-legitimate words obfuscating electronic messages according to the plurality of rules, and deleting the identified non-legitimate words from the electronic messages;
wherein said plurality of rules for distinguishing non-legitimate words from legitimate words comprises at least three rules from a group of rules consisting of:
a rule specifying a maximum number of consecutive vowels in a legitimate word; a rule specifying a maximum number of consecutive consonants in a legitimate word; a rule specifying a maximum number of consecutive uses of any single character in a legitimate word; a rule specifying a maximum number of transitions between upper case letters and lower case letters in a legitimate word; a rule specifying a maximum length of a legitimate word containing numbers without punctuation; a rule specifying a maximum length of a legitimate word containing upper case letters, lower case letters and numbers; a rule specifying a maximum length of a legitimate word containing upper case letters, lower case letters, numbers and punctuation; a rule specifying a minimum number of vowels in a legitimate word; a rule specifying a minimum number of consonants in a legitimate word; a rule specifying a minimum ratio of vowels to consonants in a legitimate word; and a rule specifying a maximum ratio of vowels to consonants in a legitimate word; and
means for analyzing normalized electronic message to identify undesirable electronic messages.
16. The computer system of claim 15 further comprising:
means for identifying a language in which an identified electronic message is composed, prior to normalizing the identified electronic message according to the set of rules; and
means for normalizing the identified electronic messages according to a set of rules taking into account the identified language.
17. The computer system of claim 15 wherein the means for normalizing identified electronic messages according to a set of rules further comprises:
means for identifying look-alike characters in electronic messages; and
means for replacing identified look-alike characters with corresponding primary characters in the electronic messages.
18. The computer system of claim 15 wherein the means for analyzing normalized electronic message to identify undesirable electronic messages further comprises:
means for keeping a count of detected items to be normalized in an electronic message; and
means for using data concerning a relationship between the count of detected items and a dynamic threshold value as evidence in the identification of an electronic message as undesirable.