Troubleshooting based on log similarity

1. A method for trouble shooting based on log similarity, the method comprising:

extracting log patterns from a plurality of log files in response to having collected the plurality of log files from at least one system with troubles, each log pattern describing a regular expression to which a log message in a log file among the plurality of log files conforms;

building a pattern repository using the log patterns extracted;

mapping each of the plurality of log files to an n-dimensional vector based on the pattern repository; and

clustering a plurality of n-dimensional vectors to which each of the plurality of log files is mapped into at least one group, wherein each of the at least one group indicates one trouble type of the at least one system;

wherein the mapping each of the plurality of log files to an n-dimensional vector based on the pattern repository comprises: with respect to a log file j among the plurality of log files,

matching each line of log message k in the log file j to a corresponding log pattern p k in the pattern repository;

transforming the log file j into a sequence f j of the corresponding log pattern p k ; and

mapping the sequence f j to an n-dimensional vector;

wherein dimension n of the n-dimensional vector is proportional to the amount of log patterns in the pattern repository;

wherein mapping the sequence f j to an n-dimensional vector comprises: with respect to a log pattern p i in the pattern repository,

calculating an eigenvalue tfidf i,j , , wherein tfidf i,j =tf i,j ×idf i ; the eigenvalue tfidf i,j associated with the sequence f j and the log pattern p i , and the eigenvalue tfidf i,j associated with an occurrence frequency of the log pattern p i in a plurality of sequences corresponding to the plurality of log files; and

treating the eigenvalue tfidf i,j as the i th component in the n-dimensional vector to build the n-dimensional vector.

2. The method according to claim 1 , wherein the extracting log patterns from plurality of log files in response to having collected the plurality log files from the at least one system with troubles comprises: with respect to a current log file among the plurality of log files,

extracting the log patterns from the log messages in the current log file.

3. The method according to claim 2 , wherein the extracting the log patterns from the log messages in the current log file comprises:

calculating the longest common subsequence of the log messages to extract the log patterns.

4. The method according to claim 1 , wherein the eigenvalue tfidf i,j is associated with the term frequency tf i,j of the log pattern p i and with the inverse document frequency idf i of the log pattern p i .

5. The method according to claim 4 , wherein

tf

i

,

j

=

t

i

,

j

∑

u

⁢

t

u

,

j

,

where t i,j represents an occurrence number of the log pattern p i in the sequence f j ; Σ u t u,j represents a sum of occurrence numbers of all log patterns in the pattern repository in the sequence f j ; and

idf

i

=

log

⁢



F



1

+

{

v

⁢

:

⁢

⁢

p

i

∈

f

v

}

,

where |F| represents the amount of the plurality of log files, and {v:p i εf v } represents the amount of sequences comprising the log pattern p i .

6. The method according to claim 1 , wherein the building a pattern repository using the log patterns comprises:

in response to an occurrence frequency of a log pattern p among the log patterns exceeding a predefined threshold, adding the log pattern p into the pattern repository.

7. The method according to claim 6 , further comprising

calculating similarity between a new log file and the plurality of vectors in at least one group in response to having received the new log file from a system;

treating a failure type indicated by a group to which a vector with the highest similarity belongs as a failure type of the system;

mapping the new log file to the n-dimensional vector; and

calculating the similarity between the n-dimensional vector and the plurality of vectors.

8. An apparatus for trouble shooting based on log similarity, comprising:

a processor;

memory in communication with the processor;

a log module, via the processor and memory, extracting log patterns from a plurality of log files in response to having collected the plurality of log files from at least one system with troubles, each log pattern describing a regular expression to which a log message in a log file among the plurality of log files conforms;

a building module, via the processor and memory, building a pattern repository using the log patterns extracted;

a mapping module, via the processor and memory, mapping each of the plurality of log files to an n-dimensional vector based on the pattern repository; and

a clustering module, via the processor and memory, clustering a plurality of n-dimensional vectors to which each of the plurality of log files is mapped into at least one group, wherein each of the at least one group indicates one trouble type of the at least one system;

wherein the mapping module is configured to, with respect to a log file j among the plurality of log files, match each line of log message k in the log file j to a corresponding log pattern p k in the pattern repository;

transform the log file j into a sequence f j of the corresponding log pattern p k ; and

map the sequence f j to an n-dimensional vector;

wherein dimension n of the n-dimensional vector is proportional to the amount of log patterns in the pattern repository;

wherein the mapping module is configured to, with respect to a log pattern p i in the pattern repository, calculate an eigenvalue tfidf i,j wherein tfidf i,j =tf i,j ×idf i ; the eigenvalue tfidf i,j associated with the sequence f j and the log pattern p i , and the eigenvalue tfidf i,j associated with an occurrence frequency of the log pattern p i in multiple sequences corresponding to the plurality of log files; and

further configured to treat the eigenvalue tfidf i,j as the i th component in the n-dimensional vector to build the n-dimensional vector.

9. The apparatus according to claim 8 , wherein the extracting module is further configured to

extract the log patterns from the log messages in the current log file.

10. The apparatus according to claim 9 , wherein the extracting module is further configured to:

calculate the longest common subsequence of the log messages so as to extract the log patterns.

11. The apparatus according to claim 8 , wherein the eigenvalue tfidf i,j is associated with the term frequency tf i,j of the log pattern p i and with the inverse document frequency idf i of the log pattern p i .

12. The apparatus according to claim 11 , wherein

tf

i

,

j

=

t

i

,

j

∑

u

⁢

t

u

,

j

,

where t i,j represents an occurrence number of the log pattern p i in the sequence f j , Σ u t u,j represents a sum of occurrence numbers of all log patterns in the pattern repository in the sequence f j ; and

idf

i

=

log

⁢



F



1

+

{

v

⁢

:

⁢

⁢

p

i

∈

f

v

}

,

where |F| represents the amount of the plurality of log files, and {v:p i εf v } represents the amount of sequences comprising the log pattern p i .

13. The apparatus according to claim 9 , wherein the building module is further configured to, in response to an occurrence frequency of a log pattern p among the log patterns exceeding a predefined threshold, add the log pattern p into the pattern repository.

14. The apparatus as claimed in claim 9 is further configured to calculate similarity between a new log file and the plurality of vectors in at least one group in response to having received the new log file from a system;

treat a failure type indicated by a group to which a vector with the highest similarity belongs as a failure type of the system;

map the new log file to the n-dimensional vector based on the pattern repository; and

calculate the similarity between the n-dimensional vector and the plurality of vectors.