System and method for elimination of spam in a data stream according to information density
A method for elimination of spam in a data stream according to information density, includes receiving, by a computing device, a stream of messages. The method includes directing, by the computing device, the stream into at least one buffer. The method includes repeatedly compressing, by the computing device, data in the buffer using a lossless compression algorithm. The method includes identifying, by the computing device, at least one first message in the buffer as spam, by determining that the at least one first message has been compressed below a threshold level.
1. A method for elimination of spam in a data stream according to information density, the method comprising:
receiving, by a computing device, a stream of messages;
directing, by the computing device, the stream into at least one buffer;
repeatedly compressing, by the computing device, data in the buffer using a lossless compression algorithm; and
identifying, by the computing device, at least one first message in the buffer as spam, by determining that the at least one first message has been compressed below a threshold level;
wherein determining that the at least one first message has been compressed below the threshold level further comprises determining that the ratio of the compressed size of the at least one first message to the uncompressed size of the at least one first message is smaller than a threshold number.
2. A method according to claim 1 , wherein receiving further comprises receiving a stream of messages containing textual data.
3. A method according to claim 2 , wherein compressing further comprises:
determining that a term in the buffer has been encountered previously;
mapping the term to a single character representation; and
replacing the term in the buffer with the single character representation.
4. A method according to claim 1 further comprising deleting the at least one first message.
5. A method according to claim 1 further comprising storing the at least one first message in a spam folder.
6. A method according to claim 1 further comprising identifying an account from which the at least one first message originated.
7. A method according to claim 6 further comprising publishing the account to a user of the computing device.
8. A method according to claim 6 further comprising blocking the reception of future messages from the account.
9. A method according to claim 1 further comprising determining that at least one second message in the buffer is not spam.
10. A method according to claim 9 wherein determining that the at least one second message is not spam further comprises:
maintaining in memory accessible to the computing device a datum representing a quantity of data received in the stream;
determining that the computing device has received a greater quantity of data in the stream than the quantity represented by the datum since receiving the at least one second message; and
determining that the at least one second message has not compressed below the threshold level.
11. A method according to claim 9 wherein determining that the at least one second message is not spam further comprises:
maintaining in memory accessible to the computing device a datum representing a number of compression cycles;
determining that the computing device has performed a greater number of compression cycles than the datum since receiving the at least one second message; and
determining that the at least one second message has not compressed below the threshold level.
12. A method according to claim 9 further comprising providing the at least one second message to a user of the computing device.
13. A system for elimination of spam in a data stream according to information density, the system comprising:
a computing device;
at least one receiver, executing on the computing device, and configured to receive a stream of messages and direct the stream into at least one buffer;
at least one compressor, executing on the computing device, and configured to repeatedly compress data in the buffer using a lossless compression algorithm; and
at least one classifier, executing on the computing device and configured to identify at least one message in the buffer as spam by determining that the at least one first message has been compressed below a threshold level;
wherein determining that the at least one first message has been compressed below the threshold level further comprises determining that the ratio of the compressed size of the at least one first message to the uncompressed size of the at least one first message is smaller than a threshold number.