Tool-specific alerting rules based on abnormal and normal patterns obtained from history logs
A computer-implemented method is presented for automatically generating alerting rules. The method includes identifying, via offline analytics, abnormal patterns and normal patterns from history logs based on machine learning, statistical analysis and deep learning, the history logs stored in a history log database, automatically generating the alerting rules based on the identified abnormal and normal patterns, and transmitting the alerting rules to an alerting engine for evaluation. The method further includes receiving a plurality of online log messages from a plurality of computing devices connected to a network, augmenting the plurality of online log messages, and extracting information from the plurality of augmented online log messages to be provided to the alerting engine, the alerting engine configured to approve and enforce the alerting rules automatically generated by the offline analytics processing.
1. A computer-implemented method comprising:
collecting history logs until a collection threshold is met;
identifying, via offline analytics, abnormal patterns and normal patterns from the history logs based on machine learning, statistical analysis and deep learning, the history logs stored in a history log database;
feeding the abnormal patterns and the normal patterns to a subject matter expert (SME) validation module to make revisions or corrections to the abnormal patterns and the normal patterns;
automatically generating alerting rule candidates based on the identified abnormal and normal patterns detected in the history logs;
transmitting the alerting rule candidates to an alerting engine to assess whether the generated alerting rule candidates are accepted or discarded;
receiving a plurality of online log messages from a plurality of computing devices connected to a network;
augmenting the plurality of online log messages; and
extracting information from the plurality of augmented online log messages to be provided to the alerting engine, the alerting engine configured to approve and enforce the alerting rule candidates accepted by the alerting engine,
wherein the collection threshold is based on a total number of the history logs, a traffic type associated with each of the total number of the history logs, and a percentage of the total number of the history logs being of a certain traffic type.
2. The method of claim 1 , wherein the history logs are clustered into a plurality of clusters and analyzed to determine normal logs and abnormal logs.
3. The method of claim 2 , wherein rare logs are identified based on predetermined policies related to the abnormal logs.
4. The method of claim 2 , wherein the normal logs are exposed to log template mining techniques to create log templates.
5. The method of claim 4 , wherein variables are extracted from the log templates, the variables are identified by variable type, and feature extraction is performed on the variables.
6. The method of claim 1 , wherein the corrections to the abnormal patterns and the normal patterns by the SME validation module include filling in blank items.
7. The method of claim 1 , wherein the generated alerting rule candidates are provided to a knowledge database constructed from log message training data.
8. A non-transitory computer-readable storage medium comprising a computer-readable program executed on a processor in a data processing system, wherein the computer-readable program when executed on the processor causes a computer to perform the steps of:
collecting history logs until a collection threshold is met;
identifying, via offline analytics, abnormal patterns and normal patterns from the history logs based on machine learning, statistical analysis and deep learning, the history logs stored in a history log database;
feeding the abnormal patterns and the normal patterns to a subject matter expert (SME) validation module to make revisions or corrections to the abnormal patterns and the normal patterns;
automatically generating alerting rule candidates based on the identified abnormal and normal patterns detected in the history logs;
transmitting the alerting rule candidates to an alerting engine to assess whether the generated alerting rule candidates are accepted or discarded;
receiving a plurality of online log messages from a plurality of computing devices connected to a network;
augmenting the plurality of online log messages; and
extracting information from the plurality of augmented online log messages to be provided to the alerting engine, the alerting engine configured to approve and enforce the alerting rule candidates accepted by the alerting engine,
wherein the collection threshold is based on a total number of the history logs, a traffic type associated with each of the total number of the history logs, and a percentage of the total number of the history logs being of a certain traffic type.
9. The non-transitory computer-readable storage medium of claim 8 , wherein the history logs are clustered into a plurality of clusters and analyzed to determine normal logs and abnormal logs.
10. The non-transitory computer-readable storage medium of claim 9 , wherein rare logs are identified based on predetermined policies related to the abnormal logs.
11. The non-transitory computer-readable storage medium of claim 9 , wherein the normal logs are exposed to log template mining techniques to create log templates.
12. The non-transitory computer-readable storage medium of claim 11 , wherein variables are extracted from the log templates, the variables are identified by variable type, and feature extraction is performed on the variables.
13. The non-transitory computer-readable storage medium of claim 8 , wherein the corrections to the abnormal patterns and the normal patterns by the SME validation module include filling in blank items.
14. The non-transitory computer-readable storage medium of claim 8 , wherein the generated alerting rule candidates are provided to a knowledge database constructed from log message training data.
15. A system comprising:
a memory; and
one or more processors in communication with the memory configured to:
collect history logs until a collection threshold is met;
identify, via offline analytics, abnormal patterns and normal patterns from the history logs based on machine learning, statistical analysis and deep learning, the history logs stored in a history log database;
feed the abnormal patterns and the normal patterns to a subject matter expert (SME) validation module to make revisions or corrections to the abnormal patterns and the normal patterns;
automatically generate alerting rule candidates based on the identified abnormal and normal patterns detected in the history logs;
transmit the alerting rule candidates to an alerting engine to assess whether the generated alerting rule candidates are accepted or discarded;
receive a plurality of online log messages from a plurality of computing devices connected to a network;
augment the plurality of online log messages; and
extract information from the plurality of augmented online log messages to be provided to the alerting engine, the alerting engine configured to approve and enforce the alerting rule candidates accepted by the alerting engine,
wherein the collection threshold is based on a total number of the history logs, a traffic type associated with each of the total number of the history logs, and a percentage of the total number of the history logs being of a certain traffic type.
16. The system of claim 15 , wherein the history logs are clustered into a plurality of clusters and analyzed to determine normal logs and abnormal logs.
17. The system of claim 16 , wherein rare logs are identified based on predetermined policies related to the abnormal logs.
18. The system of claim 16 , wherein the normal logs are exposed to log template mining techniques to create log templates.
19. The system of claim 18 , wherein variables are extracted from the log templates, the variables are identified by variable type, and feature extraction is performed on the variables.
20. The system of claim 15 , wherein the corrections to the abnormal patterns and the normal patterns by the SME validation module include filling in blank items.