Systems and methods for providing a content item database and identifying content items
View Patent ↗Systems and methods are provided for identifying unsolicited or unwanted electronic communications, such as spam. The disclosed embodiments also encompass systems and methods for selecting content items from a content item database. Consistent with certain embodiments, computer-implemented systems and methods may use a clustering based statistical content matching anti-spam algorithm to identify and filter spam. Such a anti-spam algorithm may be implemented to determine a degree of similarity between an incoming e-mail with a collection of one or more spam e-mails stored in a database. If the degree of similarity exceeds a predetermined threshold, the incoming e-mail may be classified as spam. Further, in accordance with other embodiments, systems and methods may be provided to determine a degree of similarity between a query or search string from a user and content items stored in a database. If the degree of similarity exceeds a predetermined threshold, the content item from the database may be identified as a content item that matches the query or search string provided by the user.
1. A computer-implemented method of selecting a content item from a content item database, the method comprising:
accessing an incoming query from a memory device;
creating a first set of tokens from the incoming query;
computing a first total as a number of tokens in the first set of tokens;
accessing a second set of tokens corresponding to the content item stored in the content item database;
determining, with at least one processor, a degree of similarity by:
determining a minimum of a first count and a second count, the first count being a count of a unique token in the first set of tokens and the second count being a count of the unique token in the second set of tokens;
computing a randomized easy signature by:
selecting a set of most frequent tokens from the second set of tokens;
computing a second total as a number of tokens in the selected set of most frequent tokens;
randomly selecting a sub-set of tokens from the selected set of most frequent tokens;
determining a number of common tokens based on a minimum of the first count and a third count, the third count being a count of the unique token in the randomly selected sub-set of tokens; and
determining the randomized easy signature as a ratio of the number of common tokens and a sum of the first total and the second total; and
selecting, with the at least one processor, the content item as matching the incoming query if the degree of similarity exceeds a predetermined threshold.
2. The computer-implemented method of claim 1 , wherein the incoming query comprises at least one of a search string and a user query.
3. The computer-implemented method of claim 1 , wherein creating the first set of tokens comprises:
processing the incoming query by changing an upper-case letter into a lower-case letter and removing a space; and
creating the first set of tokens from the processed incoming query, each token having a predetermined length and overlapping a previous token by including one or more characters from the previous token.
4. The computer-implemented method of claim 1 , wherein determining the degree of similarity further comprises computing an average randomized easy signature by performing the steps of:
computing a plurality of randomized easy signatures; and
averaging the plurality of randomized easy signatures.
5. The computer-implemented method of claim 3 , wherein the incoming query comprises a title of the content item.
6. The computer-implemented method of claim 3 , wherein the incoming query comprises a list of keywords corresponding to the content item.
7. The computer-implemented method of claim 3 , wherein the incoming query comprises a body of the content item.
8. The computer-implemented method of claim 3 , wherein the predetermined length of each token is three.
9. A computer-implemented system of selecting a content item from a content item database, the system comprising:
a content item database which stores a plurality of content items;
a server which performs offline processing, the offline processing comprising:
accessing a content item from the content item database;
creating a first set of tokens from the content item;
calculating a first total as a number of tokens in first set of tokens; and
storing the first set of tokens and the first total; and
a client which performs online processing, the online processing comprising:
receiving an incoming query;
creating a second set of tokens from the incoming query;
calculating a second total as a number of tokens in the second set of tokens;
accessing the first set of tokens and the first total corresponding to one of the plurality of content items in the content item database;
determining a number of common tokens based on a minimum of a first count and a second count, the first count being a count of each unique token in the first set of tokens and the second count being a count of the each unique token in the second set of tokens;
computing a randomized easy signature by:
selecting a set of most frequent tokens from the second set of tokens;
computing a second total as a number of tokens in the selected set of most frequent tokens;
randomly selecting a sub-set of tokens from the selected set of most frequent tokens;
determining a number of common tokens based on a minimum of the first count and a third count, the third count being a count of the unique token in the randomly selected sub-set of tokens; and
determining the randomized easy signature as a ratio of the number of common tokens and a sum of the first total and the second total; and
designating the content item in the content item database as matching the incoming query when the easy signature exceeds a predetermined threshold.
10. The computer-implemented system of claim 9 , wherein the incoming query comprises at least one of a search string and a user query.
11. The computer-implemented system of claim 9 , wherein the content item comprises at least one of an e-mail, an instant message, a chat message, a text messages, a SMS message, a paging communication, a blog post, and a news item.
12. The computer-implemented system of claim 9 , wherein creating a first set of tokens comprises:
processing the content item by changing an upper-case letter into a lower-case letter and removing a space; and
creating the first set of tokens from the processed content item, each token having a predetermined length and overlapping a previous token by including one or more characters from the previous token.
13. A computer program product comprising executable instructions tangibly embodied in a non-transitory computer-readable medium which, when executed by at least one processor, perform a method of selecting a content item, the method comprising:
accessing an incoming query from a memory device;
creating a first set of tokens from the incoming query;
computing a first total as a number of tokens in the first set of tokens;
accessing a second set of tokens, corresponding to the content item stored in the content item database;
determining a degree of similarity by:
determining a minimum of a first count and a second count, the first count being a count of a unique token in the first set of tokens and the second count being a count of the unique token in the second set of tokens; and
computing a randomized easy signature by:
selecting a set of most frequent tokens from the second set of tokens;
computing a second total as a number of tokens in the selected set of most frequent tokens;
randomly selecting a sub-set of tokens from the selected set of most frequent tokens;
determining a number of common tokens based on a minimum of the first count and a third count, the third count being a count of the unique token in the randomly selected sub-set of tokens; and
determining the randomized easy signature as a ratio of the number of common tokens and a sum of the first total and the second total; and
identifying the content item as matching the incoming query if the degree of similarity exceeds a predetermined threshold.
14. The computer program product of claim 13 , wherein the incoming query comprises at least one of a search string and a user query.
15. The computer program product of claim 13 , wherein the content item comprises at least one of an e-mail, an instant message, a chat message, a text messages, a SMS message, a paging communication, a blog post, and a news item.
16. The computer program product of claim 13 , wherein the method performed by the at least one processor further comprises:
computing an average randomized easy signature by performing the following:
computing a plurality of randomized easy signatures; and
averaging the plurality of randomized easy signatures.