Method, apparatus and article of manufacture for categorizing computerized messages into categories
There is disclosed herein techniques for categorizing computerized messages into categories. In one embodiment, there is disclosed a method. The method comprising performing an analysis of one or more computerized messages that includes identifying a set of discriminatory tokens in the one or more computerized messages that are representative of a category and determining for each discriminatory token a respective weight by which the token describes the category. The method also comprises determining a similarity between a computerized message and the category based on the content of the computerized message, the set of discriminatory tokens and the respective weights. The method further comprises classifying the computerized message as belonging to the category upon determining that the computerized message and the category are similar.
1. A method comprising steps of:
performing an analysis of one or more computerized messages, wherein performing the analysis comprises identifying a set of discriminatory tokens in the one or more computerized messages that are representative of a category and determining for each discriminatory token a respective weight by which the token describes the category;
determining a similarity between a computerized message and the category based on the content of the computerized message, the set of discriminatory tokens and the respective weights; and
classifying the computerized message as belonging to the category upon determining that the computerized message and the category are similar, wherein classifying the message comprises:
tokenizing and normalizing the message M, M={(m 1 , n 1 ) . . . , (m k , n k )}, where m i , n i is a token and the number of times the token appears in M;
computing a set intersection between the tokens of M and each topic-model centroid C i ; and
assigning M to the category whose set intersection is largest,
wherein the steps are performed by at least one processing device comprising a processor coupled to a memory.
2. The method as claimed in claim 1 , wherein the one or more computerized messages comprise log messages.
3. The method as claimed in claim 1 , wherein the one or more computerized messages derive from events relating to resources in an Information Technology (IT) infrastructure.
4. The method as claimed in claim 1 , wherein assigning M to the category whose set intersection is the largest comprises assigning M to the category C that maximizes the dot product between M and C.
5. The method as claimed in claim 1 , wherein performing the analysis further comprises separating content of the one or more computerized messages into a plurality of respective tokens such that the set of discriminatory tokens can be identified.
6. The method as claimed in claim 1 , wherein performing the analysis further comprises removing non-discriminatory tokens from the one or more computerized messages such that tokens deemed to be common cannot be considered as being representative of the category.
7. The method as claimed in claim 1 , wherein performing the analysis further comprises detecting an instance of a discriminatory token in the one or more computerized messages and replacing the instance of the discriminatory token with a generalized version of the discriminatory token.
8. The method as claimed in claim 7 , wherein performing the analysis further comprises converting the tokens in the one or more computerized messages to one of lower or upper case except those discriminatory tokens that have been generalized and exist in the other one of the lower and upper case.
9. The method as claimed in claim 1 , wherein performing the analysis further comprises treating each computerized message as a sentence, grouping all messages that are generated by a single device and belong to a single topic as a single text document, grouping together all documents that belong to a single topic across one or more devices, and computing document-by-term matrix using bag-of-words, normalized term frequency and counts representations.
10. The method as claimed in claim 1 , wherein performing the analysis further comprises utilizing a topic model technique to facilitate production of a single topic vector whose components include a set of discriminatory tokens identified as being representative of the category and respective weights determined for each discriminatory token.
11. The method as claimed in claim 10 , wherein utilizing the topic model technique produces multiple topic vectors from which the single topic vector is produced.
12. The method as claimed in claim 10 , wherein utilizing the topic model technique includes at least one of applying a latent Dirichlet allocation with counts document-by-term matrix representation to compute one topic vector, applying a non-negative matrix factorization with bag-of-words to compute a rank-I factorization, applying non-negative matrix factorization and normalized term-frequency representation to compute a rank-I factorization, and applying a latent semantic indexing technique.
13. The method as claimed in claim 10 , wherein performing the analysis further comprises selecting one or more of the highest weighted tokens whose combined weight does not exceed a fraction of the total weight and normalizing such that the length of the topic vector corresponds to a defined value.
14. The method as claimed in claim 10 , wherein performing the analysis further comprises representing the topic vector as a hash table keyed by the discriminatory tokens and in which values are represented by the weights.
15. The method as claimed in claim 1 , wherein determining the similarity between the computerized message and the category comprises determining for the computerized message a set of tokens and a number of times the respective tokens appear in the computerized message.
16. The method as claimed in claim 15 , wherein determining the similarity between the computerized message and the category comprises determining the set intersection between the tokens of the computerized message and the category.
17. The method as claimed in claim 16 , wherein determining the similarity between the computerized message and the category is dependent on the set intersection being largest.
18. The method as claimed in claim 16 , wherein if the set intersection between the tokens of the computerized message and the category and a set intersection between tokens of another computerized message and the category achieve a maximum intersection size, the determining of the similarity comprises projecting the computerized message onto token space of the category and computing a dot product for facilitating the classification of the computerized message to the category based on the dot product being the largest.
19. An apparatus comprising:
at least one processing device comprising a processor coupled to a memory;
said at least one processing device being configured to:
perform an analysis of one or more computerized messages, wherein performing the analysis comprises identifying a set of discriminatory tokens in the one or more computerized messages that are representative of a category and determining for each discriminatory token a respective weight by which the token describes the category;
determine a similarity between a computerized message and the category based on the content of the computerized message, the set of discriminatory tokens and the respective weights; and
classify the computerized message as belonging to the category upon determining that the computerized message and the category are similar, wherein classifying the message comprises:
tokenizing and normalizing the message M, M={(m 1 , n 1 ) . . . , (m k , n k )}, where m i , n i is a token and the number of times the token appears in M;
computing a set intersection between the tokens of M and each topic-model centroid C i ; and
assigning M to the category whose set intersection is largest.
20. An article of manufacture comprising a processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device to:
perform an analysis of one or more computerized messages, wherein performing the analysis comprises identifying a set of discriminatory tokens in the one or more computerized messages that are representative of a category and determining for each discriminatory token a respective weight by which the token describes the category;
determine a similarity between a computerized message and the category based on the content of the computerized message, the set of discriminatory tokens and the respective weights; and
classify the computerized message as belonging to the category upon determining that the computerized message and the category are similar, wherein classifying the message comprises:
tokenizing and normalizing the message M, M={(m 1 , n 1 ) . . . , (m k , n k )}, where m i , n i is a token and the number of times the token appears in M;
computing a set intersection between the tokens of M and each topic-model centroid C i ; and
assigning M to the category whose set intersection is largest.