Signature generation using message summaries
View Patent ↗Systems and methods for processing a message are provided. A message may be processed to generate a message summary by removing or replacing certain words, phrases, sentences, punctuation, and the like. Message signatures based upon the message summary may be generated and stored in a signature database, which may be used to identify and/or classify spam messages. Subsequently received messages may be classified by signature and processed based on classification.
1. A method for processing electronic-mail messages, the method comprising:
storing information in memory regarding a message previously classified as an unsolicited message, the message including a collection of words, wherein the collection of words is not helpful for the purpose of distinguishing the message, the collection of words including a plurality of words that can be found in a dictionary;
executing instructions stored in memory, wherein execution of the instructions by a processor:
identifies from the collection of words one or more dictionary words known to be commonly occurring or known to be associated with spam,
removes the identified one or more dictionary words from the message,
replaces one or more remaining words in the message with an associated canonical equivalent to generate a resulting summary, and
generates a signature based on the resulting summary, and
storing the generated signature in memory for use in classifying and processing subsequently received messages, wherein a subsequently received message is classified based on the signature and processed based on the classification.
2. The method of claim 1 , wherein the removed words include sentences or passages purposefully inserted into the message by the creator of the unsolicited message.
3. The method of claim 1 , wherein the removed words include prepositions.
4. The method of claim 1 , wherein the removed words include articles.
5. The method of claim 1 , wherein the associated canonical equivalent includes an identifier identifying a list of one or more equivalent words.
6. The method of claim 1 , further comprising correcting any incorrectly spelled words in the message prior to removing the one or more words that are not helpful for the purpose of distinguishing the message.
7. The method of claim 1 , further comprising ranking words in the resulting summary, after the resulting summary is generated and based on at least a probability that the words occur in unsolicited messages.
8. The method of claim 1 , further comprising ranking words in the resulting summary after the resulting summary is generated and based on at least a probability the words occur in natural language.
9. The method of claim 1 , wherein the signature is generated using a transform function.
10. The method of claim 1 , further comprising storing a plurality of signatures in a database.
11. The method of claim 10 , wherein the plurality of signatures are formatted using extensible markup language (XML).
12. A method for processing electronic-mail messages, the method comprising:
receiving a message sent over a communication network, the message including a collection of words, wherein the collection of words is not helpful for the purpose of distinguishing the message, the collection of words including a plurality of words that can be found in a dictionary; and
executing instructions stored in memory, wherein execution of the instructions by a processor:
identifies from the collection of words one or more dictionary words known to be commonly occurring or known to be associated with spam, and
removes the identified one or more dictionary words from the message to generate a resulting summary,
classifies the message based on at least the resulting summary, and
processes the message based on at least the classification of the message.
13. The method of claim 12 , wherein classifying the message based on at least the resulting summary comprises:
generating one or more signatures based on at least the resulting summary; and
classifying the message based on the one or more signatures.
14. The method of claim 12 , wherein classifying the message comprises determining whether the message is an unsolicited message.
15. A system for processing a message comprising:
memory for storing information regarding a message previously classified as an unsolicited message, the message including a collection of words, wherein the collection of words is not helpful for the purpose of distinguishing the message, the collection of words including a plurality of words that can be found in a dictionary;
a message gateway including a summarization and signature generation engine stored in memory, the summarization and signature engine executable by a processor to:
identify from the collection of words one or more dictionary words known to be commonly occurring or known to be associated with spam,
remove the identified one or more dictionary words in the message,
replace one or more remaining words in the message with an associated canonical equivalent to generate a resulting summary, and
generate one or more signatures based on at least the resulting summary, wherein the one or more signatures are stored in a database configured to store information concerning the one or more signatures; and
a server configured to process subsequently received messages based on at least signature information generated by the message gateway and stored in the database.
16. The system of claim 15 , wherein the database is further configured to store information concerning a plurality of signatures, the plurality of signatures being generated based on a plurality of message summaries.
17. A non-transitory computer-readable storage medium having embodied thereon a program, the program being executable by a processor to perform a method for processing a message, the method comprising:
storing information regarding a message previously classified as an unsolicited message, the message including a collection of words, wherein the collection of words is not helpful for the purpose of distinguishing the message, the collection of words including a plurality of words that can be found in a dictionary;
identifying from the collection of words one or more dictionary words known to be commonly occurring or known to be associated with spam;
removing the identified one or more dictionary words from the message;
replacing one or more remaining words in the message with an associated canonical equivalent to generate a resulting summary;
generating a signature based on the resulting summary;
classifying a subsequently received message based on the signature; and
processing the subsequently received message based on the classification of the subsequently received message.