Machine data web
Methods and apparatus consistent with the invention provide the ability to organize and build understandings of machine data generated by a variety of information-processing environments. Machine data is a product of information-processing systems (e.g., activity logs, configuration files, messages, database records) and represents the evidence of particular events that have taken place and been recorded in raw data format. In one embodiment, machine data is turned into a machine data web by organizing machine data into events and then linking events together.
1. A method, comprising:
organizing into a plurality of events machine data from a plurality of diverse sources, wherein there is no obvious common structure shared among the machine data from the plurality of diverse sources, and wherein machine data included in each event in the plurality of events retains its original structure;
determining a time stamp for each event in the plurality of events;
performing a search on the plurality of events.
2. The method as recited in claim 1 , further comprising:
automatically identifying boundaries of events within the machine data.
3. The method as recited in claim 1 , wherein organizing machine data into a plurality of events further comprises:
automatically creating rules for identifying boundaries of events within the machine data;
storing the rules in at least one storage device;
applying the rules to the machine data to define each event in the plurality of events.
4. The method as recited in claim 1 wherein an event is comprised of at least a portion of one or more lines of data within the machine data.
5. The method as recited in claim 1 , further comprising:
extracting entities from data in an event.
6. The method as recited in claim 1 , further comprising:
segmenting data in an event into tokens.
7. The method as recited in claim 1 , further comprising:
classifying events into event types.
8. The method as recited in claim 1 , further comprising:
dynamically constructing links between events;
wherein the links represent relationships between events.
9. The method as recited in claim 1 , further comprising:
dynamically constructing links between events;
wherein the links represent relationships between events;
constructing a path by chaining event links together;
generating statistical information based on occurrences of one or more paths.
10. An apparatus, comprising:
a machine data organization subsystem, implemented at least partially in hardware, that organizes into a plurality of events machine data from a plurality of diverse sources, wherein there is no obvious common structure shared among the machine data from the plurality of diverse sources, and wherein machine data included in each event in the plurality of events retains its original structure;
a time stamp subsystem, implemented at least partially in hardware, that determines a time stamp for each event in the plurality of events;
a search subsystem, implemented at least partially in hardware, that performs a search on the plurality of events.
11. The apparatus as recited in claim 10 , further comprising:
a boundary subsystem, implemented at least partially in hardware, that automatically identifies boundaries of events within the machine data.
12. The apparatus as recited in claim 10 , wherein the organizing subsystem further comprises:
a rule subsystem, implemented at least partially in hardware, that automatically creates rules for identifying boundaries of events within the machine data;
a storing subsystem, implemented at least partially in hardware, that stores the rules in at least one storage device;
a rule application subsystem, implemented at least partially in hardware, that applies the rules to the machine data to define each event in the plurality of events.
13. The apparatus as recited in claim 10 wherein an event is comprised of at least a portion of one or more lines of data within the machine data.
14. The apparatus as recited in claim 10 , further comprising:
an extraction subsystem, implemented at least partially in hardware, that extracts entities from data in an event.
15. The apparatus as recited in claim 10 , further comprising:
a segmentation subsystem, implemented at least partially in hardware, that segments data in an event into tokens.
16. The apparatus as recited in claim 10 , further comprising:
a classification subsystem, implemented at least partially in hardware, that classifies events into event types.
17. The apparatus as recited in claim 10 , further comprising:
a link subsystem, implemented at least partially in hardware, that dynamically constructs links between events;
wherein the links represent relationships between events.
18. The apparatus as recited in claim 10 , further comprising:
a link subsystem, implemented at least partially in hardware, that dynamically constructs links between events;
wherein the links represent relationships between events;
a path subsystem, implemented at least partially in hardware, that constructs a path by chaining event links together;
a statistics subsystem, implemented at least partially in hardware, that generates statistical information based on occurrences of one or more paths.
19. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to perform:
organizing into a plurality of events machine data from a plurality of diverse sources, wherein there is no obvious common structure shared among the machine data from the plurality of diverse sources, and wherein machine data included in each event in the plurality of events retains its original structure;
determining a time stamp for each event in the plurality of events;
performing a search on the plurality of events.
20. The non-transitory computer-readable medium as recited in claim 19 , further comprising:
automatically identifying boundaries of events within the machine data.
21. The non-transitory computer-readable medium as recited in claim 19 , wherein organizing machine data into a plurality of events further comprises:
automatically creating rules for identifying boundaries of events within the machine data;
storing the rules in at least one storage device;
applying the rules to the machine data to define each event in the plurality of events.
22. The non-transitory computer-readable medium as recited in claim 19 wherein an event is comprised of at least a portion of one or more lines of data within the machine data.
23. The non-transitory computer-readable medium as recited in claim 19 , further comprising:
extracting entities from data in an event.
24. The non-transitory computer-readable medium as recited in claim 19 , further comprising:
segmenting data in an event into tokens.
25. The non-transitory computer-readable medium as recited in claim 19 , further comprising:
classifying events into event types.
26. The non-transitory computer-readable medium as recited in claim 19 , further comprising:
dynamically constructing links between events;
wherein the links represent relationships between events.
27. The non-transitory computer-readable medium as recited in claim 19 , further comprising:
dynamically constructing links between events;
wherein the links represent relationships between events;
constructing a path by chaining event links together;
generating statistical information based on occurrences of one or more paths.