Security system using automatic and scalable log pattern learning in security log analysis

1. A security system for providing computer system security, comprising:

one or more management services implemented by at least one hardware component and configured to generate security logs, the one or more management services including a security event management (SEM) service and a security information management (SIM) service for implementing a security information and event management (SIEM) service; and

a security log analysis service operatively coupled to the one or more management services, wherein the security log analysis service is configured to:

collect the security logs generated by the one or more management services;

implement an incremental learning process to generate a set of log patterns from the collected security logs;

parse the collected security logs using the set of log patterns to obtain parsed security logs; and

analyze the parsed security logs for one or more security applications;

wherein, in implementing the incremental learning process to generate the set of log patterns, the security log analysis service is further configured to:

define a first set as the training set, a second set as a set of log patterns that have been generated, and a third set as a set of logs of the training set that lack a matching pattern in the second set;

sample the third set to generate a fourth set, wherein the fourth set has a size corresponding to a parameter controlling a maximum resource requirement for the incremental learning process;

perform automatic log pattern recognition to generate a fifth set; and

perform a log filtering process based on the fifth set.

2. The security system as recited in claim 1 , wherein the SEM service is implemented using at least one hardware component selected from the group consisting of: firewall, proxy server, virtual private network, perimeter device, and combinations thereof.

3. The security system as recited in claim 1 , wherein the SIM service is implemented using at least one hardware component selected from the group consisting of: desktop, server, router, switch, application, syslog device, and combinations thereof.

4. The security system as recited in claim 1 , wherein the collected security logs are parsed as structured data.

5. The security system as recited in claim 1 , wherein the security logs include heterogeneous logs.

6. The security system as recited in claim 1 , wherein each pattern of the set of log patterns is in terms of regular expression.

7. The security system as recited in claim 1 , wherein, in performing the automatic log pattern recognition, the at least one hardware processor of the security log analysis service is further configured to:

tokenize logs of the fourth set to generate tokens from the logs of the fourth set;

apply a similarity measurement on the logs to capture similarities among the logs of the fourth set;

implement a hierarchical clustering algorithm to generate a log cluster hierarchy for the logs of the fourth set;

align the logs of the fourth set within each cluster associated with a given level of the log cluster hierarchy;

conduct log motif discovery on the aligned logs to find log motifs; and

perform pattern recognition from the log motifs by recognizing one or more log fields.

8. The security system as recited in claim 1 , wherein, in performing the log filtering process, the security log analysis service is further configured to:

determine if the fifth set is empty;

in response to determining that the fifth set is empty:

update the third set by subtracting the fourth set from the third set;

in response to determining that the updated third set is empty, output the second set; and

in response to determining that the updated third set includes at least one log,

update the fourth set by sampling the updated third set, and updating the fifth set by performing automatic pattern recognition using the updated fourth set; and

in response to determining that the fifth set includes at least one log pattern:

update the second set by adding the fifth set to the second set;

update the third set with logs identified in the third set that lack a matching pattern in the second set;

in response to determining that the updated third set is empty, output the updated second set; and

in response to determining that the updated third set includes at least one log, update the fourth set by sampling the updated third set, and update the fifth set by performing automatic pattern recognition using the updated fourth set.

9. The security system as recited in claim 8 , wherein the security log analysis service is further configured to use a log parser to identify the logs in the third set that lack a matching pattern in the second set.

10. The security system as recited in claim 1 , wherein the security log analysis service is further configured to archive the security logs prior to the analysis.

11. The security system as recited in claim 1 , wherein the one or more security applications include a log-based fault management application to detect abnormal logs that do not match any log pattern of the set of the log patterns.

12. A computer-implemented method for providing computer system security, comprising:

collecting, by a processor operatively coupled to a memory, security logs generated by one or more management services of a security system, the one or more management services including a security event management (SEM) service and a security information management (SIM) service for implementing a security information and event management (SIEM) service;

implementing, by the processor, an incremental learning process to generate a set of log patterns from the collected security logs;

parsing, by the processor, the collected security logs using the set of log patterns to obtain parsed security logs; and

analyzing, by the processor, the parsed security logs for one or more security applications;

wherein implementing the incremental learning process to generate the set of log patterns further comprises:

defining a first set as the training set, a second set as a set of log patterns that have been generated, and a third set as a set of logs of the training set that lack a matching pattern in the second set;

sampling the third set to generate a fourth set, wherein the fourth set has a size corresponding to a parameter controlling a maximum resource requirement for the incremental learning process;

performing automatic log pattern recognition to generate a fifth set; and

performing a log filtering process based on the fifth set.

13. The method as recited in claim 12 , wherein performing the automatic log pattern recognition further comprises:

tokenizing logs of the fourth set to generate tokens from the logs of the fourth set;

applying a similarity measurement on the logs to capture similarities among the logs of the fourth set;

implementing a hierarchical clustering algorithm to generate a log cluster hierarchy for the logs of the fourth set;

aligning the logs of the fourth set within each cluster associated with a given level of the log cluster hierarchy;

conducting log motif discovery on the aligned logs to find log motifs; and

performing pattern recognition from the log motifs by recognizing one or more log fields.

14. The method as recited in claim 12 wherein performing the log filtering process further comprises:

determining if the fifth set is empty;

in response to determining that the fifth set is empty:

updating the third set by subtracting the fourth set from the third set;

in response to determining that the updated third set is empty, outputting the second set; and

in response to determining that the updated third set includes at least one log,

updating the fourth set by sampling the updated third set, and updating the fifth set by performing automatic pattern recognition using the updated fourth set; and

in response to determining that the fifth set includes at least one log pattern:

updating the second set by adding the fifth set to the second set;

updating the third set with logs identified in the third set that lack a matching pattern in the second set;

in response to determining that the updated third set is empty, outputting the updated second set; and

in response to determining that the updated third set includes at least one log, updating the fourth set by sampling the updated third set, and updating the fifth set by performing automatic pattern recognition using the updated fourth set.

15. The method as recited in claim 14 , further comprising using, by the processor, a log parser to identify the logs in the third set that lack a matching pattern in the second set.

16. The method as recited in claim 12 , further comprising archiving, by the processor, the parsed security logs prior to the analysis.

17. A computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method for providing computer system security, comprising:

collecting security logs generated by one or more management services of a security system, the one or more management services including a security event management (SEM) service and a security information management (SIM) service for implementing a security information and event management (SIEM) service;

implementing an incremental learning process to generate a set of log patterns from the collected security logs;

parsing the collected security logs using the set of log patterns to obtain parsed security logs; and

analyzing the parsed security logs for one or more security applications;

wherein implementing the incremental learning process to generate the set of log patterns further comprises:

defining a first set as the training set, a second set as a set of log patterns that have been generated, and a third set as a set of logs of the training set that lack a matching pattern in the second set;

sampling the third set to generate a fourth set, wherein the fourth set has a size corresponding to a parameter controlling a maximum resource requirement for the incremental learning process;

performing automatic log pattern recognition to generate a fifth set; and performing a log filtering process based on the fifth set.

18. The computer program product as recited in claim 17 , wherein performing the automatic log pattern recognition further comprises:

tokenizing logs of the fourth set to generate tokens from the logs of the fourth set;

applying a similarity measurement on the logs to capture similarities among the logs of the fourth set;

implementing a hierarchical clustering algorithm to generate a log cluster hierarchy for the logs of the fourth set;

aligning the logs of the fourth set within each cluster associated with a given level of the log cluster hierarchy;

conducting log motif discovery on the aligned logs to find log motifs; and

performing pattern recognition from the log motifs by recognizing one or more log fields.

19. The computer program product as recited in claim 17 , wherein performing the log filtering process further comprises:

determining if the fifth set is empty;

in response to determining that the fifth set is empty:

updating the third set by subtracting the fourth set from the third set;

in response to determining that the updated third set is empty, outputting the second set; and

in response to determining that the updated third set includes at least one log, updating the fourth set by sampling the updated third set, and updating the fifth set by performing automatic pattern recognition using the updated fourth set; and

in response to determining that the fifth set includes at least one log pattern:

updating the second set by adding the fifth set to the second set;

updating the third set with logs identified in the third set that lack a matching pattern in the second set;

in response to determining that the updated third set is empty, outputting the updated second set; and

in response to determining that the updated third set includes at least one log, updating the fourth set by sampling the updated third set, and updating the fifth set by performing automatic pattern recognition using the updated fourth set.

20. The computer program product as recited in claim 19 , wherein the method further includes using a log parser to identify the logs in the third set that lack a matching pattern in the second set.