Real-time categorization of log events

1. A method for categorizing a real-time log event, the method comprising:

computing a Term Frequency-Inverse Document Frequency (TF-IDF) matrix of a log corpus based on a number of pre-existing log events in the log corpus and a number of words in the log corpus;

computing a Term Frequency-Inverse Document Frequency (TF-IDF) vector for the real-time log event based on a pre-calculated TF-IDF matrix of the log corpus and a number of new words in the real-time log event, wherein the log corpus comprises one or more pre-existing log events, and wherein the real-time log event is indicative of an error message;

generating a cluster model based on the TF-IDF matrix, wherein the cluster model is indicative of a number of clusters corresponding to the log corpus, and wherein a cluster is indicative of a log category;

determining a centroid matrix of the log corpus based on the number of clusters in the cluster model and the number of words in the log corpus;

calculating a cluster radius and a silhouette width of each cluster, wherein the cluster radius of a cluster is calculated based on a distance between a cluster centroid of the cluster and a farthest point in the cluster; and wherein the silhouette width of the cluster is indicative of compactness of the cluster;

determining a silhouette threshold for each cluster based on the corresponding cluster radius and the corresponding silhouette width;

calculating a distance between the TF-IDF vector and the cluster centroid of each cluster in the log corpus;

identifying, from amongst the clusters, a cluster having a closest cluster centroid based on the distance between the TF-IDF vector and the cluster centroid of each of the clusters, wherein the closest cluster centroid is a cluster centroid closest to the TF-IDF vector; and

categorizing the real-time log event into one or more log categories based on a comparison of the distance between the TF-IDF vector and the closest cluster centroid with a pre-determined silhouette threshold corresponding to the cluster with the closest cluster centroid.

2. The method as claimed in claim 1 further comprising:

receiving the real-time log event from one or more log sources; and

processing the real-time log event to remove insignificant data from the real-time log event, wherein the insignificant data includes timestamps, digits, and special characters.

3. The method as claimed in claim 1 further comprising determining a centroid matrix for the real-time log event by adapting a pre-determined centroid matrix of the log corpus based on the number of new words in the real-time log event, wherein the pre-determined centroid matrix is determined based on the number of clusters in the log corpus.

4. The method as claimed in claim 1 , wherein the one or more log categories include a pre-existing log category corresponding to the cluster and a new log category.

5. The method as claimed in claim 4 , wherein, when the distance between the TF-IDF vector and the closest cluster centroid of the cluster is greater than the pre-determined silhouette threshold corresponding to the cluster, the real-time log event is categorized as the new log category.

6. The method as claimed in claim 4 , wherein, when the distance between the TF-IDF vector and the closest cluster centroid of the cluster is less than the pre-determined silhouette threshold corresponding to the cluster, the real-time log event is categorized into the pre-existing log category.

7. The method as claimed in claim 1 , wherein the method further comprising:

receiving the log corpus from one or more log sources, wherein the log corpus comprises one or more pre-existing log events;

processing the log corpus to remove insignificant data from each of the one or more pre-existing log events, wherein the insignificant data includes timestamps, digits, and special characters.

8. The method as claimed in claim 1 , wherein the cluster model is generated based on a clustering algorithm, wherein the clustering algorithm is a spherical k-means clustering algorithm.

9. A log categorization system for categorizing a real-time log event, the log categorization system comprising:

a processor;

a clustering module coupled to the processor to,

computer a Term Frequency-Inverse Document Frequency (TF-IDF) matrix of a log corpus based on a number of pre-existing log events in the log corpus and a number of words in the log corpus;

compute a Term Frequency-Inverse Document Frequency (TF-IDF) vector for the real-time log event based on a pre-calculated TF-IDF matrix of a log corpus and a number of new words in the real-time log event, wherein the log corpus comprises one or more pre-existing log events, and wherein the real-time log event is indicative of an error message;

generate a cluster model based on the TF-IDF matrix, wherein the cluster model is indicative of the number of clusters corresponding to the log corpus, and wherein a cluster is indicative of a log category; and

determine the centroid matrix of the log corpus based on the number of clusters in the cluster model and the number of words in the log corpus;

a log categorization module coupled to the processor to,

calculate a cluster radius and a silhouette width of each cluster, wherein the cluster radius of a cluster is calculated based on a distance between a cluster centroid of the cluster and a farthest point in the cluster; and wherein the silhouette width of the cluster is indicative of compactness of the cluster;

determine a silhouette threshold for each cluster based on the corresponding cluster radius and the corresponding silhouette width;

calculate a distance between the TF-IDF vector and a cluster centroid of each cluster in the log corpus;

identify, from amongst the clusters, a cluster having a closest cluster centroid based on the distance between the TF-IDF vector and the cluster centroid of each of the clusters, wherein the closest cluster centroid is a cluster centroid closest to the TF-IDF vector; and

categorize the real-time log event into a log category based on a comparison of the distance between the TF-IDF vector and the closest cluster centroid with a pre-determined silhouette threshold corresponding to the cluster with the closest cluster centroid.

10. The log categorization system as claimed in claim 9 , wherein the log category is one of a pre-existing log category and a new log category.

11. The log categorization system as claimed in claim 9 , wherein the log categorization system further includes a log processing module coupled to the processor to:

receive the real-time log event from a log source; and

process the real-time log event to remove insignificant data from the real-time log event, wherein the insignificant data includes timestamps, digits, and special characters.

12. The log categorization system as claimed in claim 10 , wherein the log categorization module categorizes the real-time log event into the pre-existing log category when the distance between the TF-IDF vector and the closest cluster centroid is less than the pre-determined silhouette threshold corresponding to the cluster with the closest cluster centroid.

13. The log categorization system as claimed in claim 10 , wherein the log categorization module categorizes the real-time log event as the new log category when the distance between the TF-IDF vector and the closest cluster centroid is greater than the pre-determined silhouette threshold corresponding to the cluster with the closest cluster centroid.

14. The log categorization system as claimed in claim 9 , wherein the clustering module determines a centroid matrix for the real-time log event by adapting a pre-determined centroid matrix of the log corpus based on the number of new words in the real-time log event, wherein the pre-determined centroid matrix is determined based on the number of clusters in the log corpus.

15. The log categorization system as claimed in claim 9 , wherein the log processing module further:

receives the log corpus from one or more log sources, wherein the log corpus comprises one or more pre-existing log events; and

processes the log corpus to remove insignificant data from each of the one or more pre-existing log events, wherein the insignificant data includes timestamps, digits, and special characters.

16. A non-transitory computer-readable medium having embodied thereon a computer program for executing a method comprising:

computing a Term Frequency-Inverse Document Frequency (TF-IDF) matrix of a log corpus based on a number of pre-existing log events in the log corpus and a number of words in the log corpus;

computing a Term Frequency-Inverse Document Frequency (TF-IDF) vector for a log event based on a pre-calculated TF-IDF matrix of the log corpus and a number of new words in the log event, wherein the log corpus comprises one or more pre-existing log events, and wherein the log event is indicative of an error message;

generating a cluster model based on the TF-IDF matrix, wherein the cluster model is indicative of a number of clusters corresponding to the log corpus, and wherein a cluster is indicative of a log category;

determining a centroid matrix of the log corpus based on the number of clusters in the cluster model and the number of words in the log corpus;

calculating a cluster radius and a silhouette width of each cluster, wherein the cluster radius of a cluster is calculated based on a distance between a cluster centroid of the cluster and a farthest point in the cluster; and wherein the silhouette width of the cluster is indicative of compactness of the cluster;

determining a silhouette threshold for each cluster based on the corresponding cluster radius and the corresponding silhouette width;

calculating a distance between the TF-IDF vector and a cluster centroid of each cluster in the log corpus;

identifying, from amongst the clusters, a cluster having a closest cluster centroid based on the distance between the TF-IDF vector and the cluster centroid of each of the clusters, wherein the closest cluster centroid is a cluster centroid closest to the TF-IDF vector; and

categorizing the log event into one or more log categories based on a comparison of the distance between the TF-IDF vector and the closest cluster centroid with a pre-determined silhouette threshold corresponding to the cluster with the closest cluster centroid.