Multibyte heterogeneous log preprocessing

1. A method for log management, comprising:

pre-processing a plurality of heterogeneous logs, said pre-processing comprising:

performing a fixed tokenization of a plurality of heterogeneous logs based on a predefined set of symbols;

performing a flexible tokenization of the plurality of heterogeneous logs based on a user-defined set of rules;

converting timestamps in the plurality of heterogeneous logs to a single target timestamp format; and

performing structural log tokenization of the plurality of heterogeneous logs based on user-defined structural information; and

performing a log management action on the pre-processed plurality of heterogeneous logs.

2. The method of claim 1 , wherein performing fixed log tokenization comprises tokenizing according to the predefined set of symbols in both a single-byte encoding and in a multi-byte encoding.

3. The method of claim 1 , wherein performing fixed log tokenization comprises removing special characters from tokens.

4. The method of claim 1 , wherein the user-defined set of rules comprises a split rule that determines when and how a string is to be broken into substrings and a merge rule that determines when and how substrings are to be merged into a single string.

5. The method of claim 3 , wherein the user-defined set of rules comprises at least one merge rule that merges substrings that were split by the fixed tokenization.

6. The method of claim 1 , wherein flexible tokenization is performed on the output of the fixed tokenization.

7. The method of claim 1 , wherein converting timestamps comprises detecting timestamps in the plurality of heterogeneous logs based on a set of known timestamp formats.

8. The method of claim 1 , wherein the user-defined structural information includes at least one delimiter.

9. A log management system, comprising:

a fixed tokenization module configured to perform a fixed tokenization of a plurality of heterogeneous logs based on a predefined set of symbols;

a flexible tokenization module configured to perform a flexible tokenization of the plurality of heterogeneous logs based on a user-defined set of rules;

a timestamp conversion module configured to convert timestamps in the plurality of heterogeneous logs to a single target timestamp format;

a structural tokenization module configured to perform structural log tokenization of the plurality of heterogeneous logs based on user-defined structural information; and

a log analysis module configured to perform a log management action on the pre-processed plurality of heterogeneous logs.

10. The system of claim 9 , the fixed log tokenization module is further configured to tokenize according to the predefined set of symbols in both a single-byte encoding and in a multi-byte encoding.

11. The system of claim 9 , the fixed log tokenization module is further configured to remove special characters from tokens.

12. The system of claim 9 , wherein the user-defined set of rules comprises a split rule that determines when and how a string is to be broken into substrings and a merge rule that determines when and how substrings are to be merged into a single string.

13. The system of claim 12 , wherein the user-defined set of rules comprises at least one merge rule that merges substrings that were split by the fixed tokenization.

14. The system of claim 9 , the flexible log tokenization module is further configured to perform flexible tokenization on the output of the fixed tokenization.

15. The system of claim 9 , wherein the timestamp conversion module is further configured to detect timestamps in the plurality of heterogeneous logs based on a set of known timestamp formats.

16. The system of claim 9 , wherein the user-defined structural information includes at least one delimiter.