Automated anomaly detection service on heterogeneous log streams
Systems and methods are disclosed for handling log data from one or more applications, sensors or instruments by receiving heterogeneous logs from arbitrary/unknown systems or applications; generating regular expression patterns from the heterogeneous log sources using machine learning and extracting a log pattern therefrom; generating models and profiles from training logs based on different conditions and updating a global model database storing all models generated over time; tokenizing raw log messages from one or more applications, sensors or instruments running a production system; transforming incoming tokenized streams are into data-objects for anomaly detection and forwarding of log messages to various anomaly detectors; and generating an anomaly alert from the one or more applications, sensors or instruments running a production system.
1. A method for handling log data from one or more applications, sensors or instruments, comprising:
receiving heterogeneous logs from arbitrary/unknown systems or applications;
generating regular expression patterns from the heterogeneous log sources using machine learning and extracting a log pattern therefrom;
generating models and profiles from training logs based on different conditions and updating a global model database storing all models generated over time, wherein generating the models comprises generating sequence order model that extracts sequential ordering relationships between patterns and volume mode that maintains a frequency distribution of logs of each pattern and detecting unusual spikes of certain patterns and report the spikes as alerts;
tokenizing raw log messages from one or more applications, sensors or instruments running a production system;
transforming incoming tokenized streams are into data-objects for anomaly detection and forwarding of log messages to various anomaly detectors; and
generating an anomaly alert from the one or more applications, sensors or instruments running a production system.
2. The method of claim 1 , wherein each log message comprises a time stamp and text content.
3. The method of claim 1 , comprising dividing data into training logs and testing logs.
4. The method of claim 1 , wherein each anomaly violation comprises a semantic, syntactic or statistical violation.
5. The method of claim 1 , comprising generating a visual alert to a user.
6. The method of claim 1 , wherein the log pattern extraction generates a regular expression for the incoming logs using unsupervised learning.
7. The method of claim 1 , wherein the log patterns have variable fields with a wildcard pattern and each fed has a key name attributed to known patterns and to unknown fields with generic names.
8. The method of claim 1 , wherein generating the model comprises creating training profiles based on syntactic and semantic analysis.
9. The method of claim 1 , comprising generating a Content Profile Model to create a frequency profile of various values for each key in a pattern or regular expression of a category of logs.
10. The method of claim 1 , comprising performing distributed learning and incremental learning dependent on the model.
11. The method of claim 10 , wherein the global model database provides an interface to assist, and to allow for a learning service which can maintain trained profiles in a database.
12. The method of claim 1 , comprising selecting a model based on timestamp, sources, model category or based on complex queries including join, group, aggregate for grouping model categories, and aggregating models across different time ranges.
13. The method of claim 1 , comprising performing model-updates to support incremental or distributed learning processes, wherein an update of the model depends on a learning algorithm and a model profile.
14. The method of claim 1 , comprising forming a model database with a hierarchical schema and each model includes <TimeStamp, Time Range, Category, Source, Model>.
15. The method of claim 1 , comprising transforming incoming data streams into data objects for common anomaly detection tasks.
16. The method of claim 15 , comprising pushing transformed data objects to publish/subscribe module, and subscribing to the module for anomaly detection.
17. The method of claim 16 , comprising applying violation or anomaly checking to check if an incoming transformed log-stream indicates an anomaly in the system.
18. A system comprising:
a processor;
a memory;
an Internet-of-Thing (IoT) sensor providing data to the processor; and computer readable code executed by the processor to:
receiving heterogeneous logs from arbitrary/unknown systems or applications;
generating regular expression patterns from the heterogeneous log sources using machine learning and extracting a log pattern therefrom;
generating models and profiles from training lope based on different conditions and updating a global model database storing all models generated over time, wherein generating the models comprises generating sequence order model that extracts sequential ordering relationships between patterns and volume model that maintains a frequency distribution of logs of each pattern and detecting unusual spikes of certain patterns and report the spikes as alerts;
tokenizing raw log messages from one or more applications, sensors or instruments running a production system;
transforming incoming tokenized streams are into data-objects for anomaly detection and forwarding of log messages to various anomaly detectors; and
generating an anomaly alert from one or more applications, sensors or instruments running a production system.