Automatic creation of rules for identifying event boundaries in machine data
Methods and apparatus consistent with the invention provide the ability to organize and build understandings of machine data generated by a variety of information-processing environments. Machine data is a product of information-processing systems (e.g., activity logs, configuration files, messages, database records) and represents the evidence of particular events that have taken place and been recorded in raw data format. In one embodiment, machine data is turned into a machine data web by organizing machine data into events and then linking events together.
1. A method, comprising:
analyzing machine data stored in at least one storage device;
based on the machine data analysis, automatically creating rules, for different machine data sources, for identification of boundaries of events within the machine data;
organizing machine data into a plurality of events using at least one rule among the automatically created rules in order to segment the machine data into a plurality of events by determining beginning and ending of each event in the plurality of events in the machine data, each event in the plurality of events including a portion of the machine data segmented for that event, the plurality of events including both events produced from a first data source and events produced from a second data source that is different from the first data source, the machine data in one or more events produced from the first data source having a different data format than the machine data in one or more events produced from the second data source;
wherein the method is performed by one or more computing devices.
2. The method as recited in claim 1 , wherein an event is comprised of at least a portion of one or more lines of data within the machine data.
3. The method as recited in claim 1 , further comprising:
extracting entities from data in an event.
4. The method as recited in claim 1 , further comprising:
automatically identifying a machine data source using at least a portion of the machine data.
5. The method as recited in claim 1 , further comprising:
segmenting data in an event into tokens.
6. The method as recited in claim 1 , further comprising:
classifying events into event types.
7. The method as recited in claim 1 , further comprising:
constructing links between events;
wherein the links represent relationships between events.
8. The method as recited in claim 1 , further comprising:
constructing links between events;
wherein the links represent relationships between events;
constructing a path by chaining event links together;
generating statistical information based on occurrences of one or more paths.
9. The method as recited in claim 1 , wherein at least one rule among the automatically created rules identifies an event boundary as a line staring with a timestamp.
10. An apparatus, comprising:
a subsystem, implemented at least partially in hardware, that analyzes machine data stored in at least one storage device;
a subsystem, implemented at least partially in hardware, that, based on the machine data analysis, automatically creates rules, for different machine data sources, for identification of boundaries of events within the machine data;
a subsystem, implemented at least partially in hardware, that organizes machine data into a plurality of events using at least one rule among the automatically created rules in order to segment the machine data into a plurality of events by determining beginning and ending of each event in the plurality of events in the machine data, each event in the plurality of events including a portion of the machine data segmented for that event, the plurality of events including both events produced from a first data source and events produced from a second data source that is different from the first data source, the machine data in one or more events produced from the first data source having a different data format than the machine data in one or more events produced from the second data source.
11. The apparatus as recited in claim 10 , wherein an event is comprised of at least a portion of one or more lines of data within the machine data.
12. The apparatus as recited in claim 10 , further comprising:
a subsystem, implemented at least partially in hardware, that extracts entities from data in an event.
13. The apparatus as recited in claim 10 , further comprising:
a subsystem, implemented at least partially in hardware, that automatically identifies a machine data source using at least a portion of the machine data.
14. The apparatus as recited in claim 10 , further comprising:
a subsystem, implemented at least partially in hardware, that segments data in an event into tokens.
15. The apparatus as recited in claim 10 , further comprising:
a subsystem, implemented at least partially in hardware, that classifies events into event types.
16. The apparatus as recited in claim 10 , further comprising:
a subsystem, implemented at least partially in hardware, that constructs links between events;
wherein the links represent relationships between events.
17. The apparatus as recited in claim 10 , further comprising:
a subsystem, implemented at least partially in hardware, that constructs links between events;
wherein the links represent relationships between events;
a subsystem, implemented at least partially in hardware, that constructs a path by chaining event links together;
a subsystem, implemented at least partially in hardware, that generates statistical information based on occurrences of one or more paths.
18. The apparatus as recited in claim 10 , wherein at least one rule among the automatically created rules identifies an event boundary as a line staring with a timestamp.
19. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to perform:
analyzing machine data stored in at least one storage device;
based on the machine data analysis, automatically creating rules, for different machine data sources, for identification of boundaries of events within the machine data;
organizing machine data into a plurality of events using at least one rule among the automatically created rules in order to segment the machine data into a plurality of events by determining beginning and ending of each event in the plurality of events in the machine data, each event in the plurality of events including a portion of the machine data segmented for that event, the plurality of events including both events produced from a first data source and events produced from a second data source that is different from the first data source, the machine data in one or more events produced from the first data source having a different data format than the machine data in one or more events produced from the second data source.
20. The non-transitory computer-readable medium as recited in claim 19 , wherein an event is comprised of at least a portion of one or more lines of data within the machine data.
21. The non-transitory computer-readable medium as recited in claim 19 , further comprising:
extracting entities from data in an event.
22. The non-transitory computer-readable medium as recited in claim 19 , further comprising:
automatically identifying a machine data source using at least a portion of the machine data.
23. The non-transitory computer-readable medium as recited in claim 19 , further comprising:
segmenting data in an event into tokens.
24. The non-transitory computer-readable medium as recited in claim 19 , further comprising:
classifying events into event types.
25. The non-transitory computer-readable medium as recited in claim 19 , further comprising:
dynamically constructing links between events;
wherein the links represent relationships between events.
26. The non-transitory computer-readable medium as recited in claim 19 , further comprising:
constructing links between events;
wherein the links represent relationships between events;
constructing a path by chaining event links together;
generating statistical information based on occurrences of one or more paths.
27. The non-transitory computer-readable medium as recited in claim 19 , wherein at least one rule among the automatically created rules identifies an event boundary as a line staring with a timestamp.