System for decomposing events and unstructured data
A system is provided that executes artificial intelligence for unstructured data. A memory coupled to a processor that executes instructions for: a first engine using artificial intelligence (AI) to create a structured event or scraped structured event records from unstructured and semi-structured log messages; an extraction engine in communication with a managed infrastructure and the first engine, the extraction engine configured to receive managed infrastructure data; and a signaliser engine that includes one or more of a NMF engine, a k-means clustering engine and a topology proximity engine, the signaliser engine inputting a list of devices and a list a connection between components or nodes in the managed infrastructure, the signaliser engine determining one or more common characteristics and produces one or more clusters of events.
1. A system that generates an artificial intelligence (AI)) model for an unstructured data system, comprising:
a processor;
a memory coupled to the processor, the memory executing instructions for a first engine using artificial intelligence (AI) to create a structured event or scraped structured event records from unstructured and semi-structured log messages;
a visual creation tool to create labelled training data suitable to train the AI model which takes unstructured data and creates events that are converted into words and subjects to group the events into clusters that relate to failures or errors in the managed infrastructure;
an extraction engine in communication with a managed infrastructure and the first engine, the extraction engine configured to receive managed infrastructure data;
a signaliser engine that includes one or more of an NMF engine, a k-means clustering engine and a topology proximity engine, the signaliser engine inputting a list of devices and a list a connection between components or nodes in the managed infrastructure, the signaliser engine determining one or more common characteristics and produces one or more clusters of events, and
wherein the visual creation tool creates labelled training data suitable to train the AI model used to take the unstructured data and create the events, and visual creation of the labelled training data provides: (i) identification of alert templates from the unstructured data; (ii) a visual mechanism for configuration of a data ingestion process, the visual creation tool using the alert templates to map tokens from the unstructured data to specific event attributes and prompts users with suggested mappings for any remaining alert templates, the system learning from user interaction.
2. The system of claim 1 , further comprising:
a manager that provides for the occurred event one or more of: an indication of: what the event was, what generated the event, a source of the event; and a host device or host application that generated the event.
3. The system of claim 1 , wherein an event is from one or more of: a database; and an application.
4. The system of claim 1 , wherein a plurality of different attributes describes the event to denote a problem that has occurred.
5. The system of claim 1 , wherein a second field represents a source of the message.
6. The system of claim 5 , wherein a third field includes severity information.
7. The system of claim 1 , wherein everything in a file of a stream of managed infrastructure data has the same format.
8. The system of claim 1 , wherein each of a row of alerts in a set of rows represents an unstructured message.
9. The system of claim 8 , wherein each row in the set of rows represents a different type of message.
10. The system of claim 1 , wherein each record is different.