Uniform storage and search of events derived from machine data from different sources
Methods and apparatus consistent with the invention provide the ability to organize and build understandings of machine data generated by a variety of information-processing environments. Machine data is a product of information-processing systems (e.g., activity logs, configuration files, messages, database records) and represents the evidence of particular events that have taken place and been recorded in raw data format. In one embodiment, machine data is turned into a machine data web by organizing machine data into events and then linking events together.
1. A method, comprising:
analyzing machine data stored in at least one storage device in order to segment the machine data into a plurality of events by determining beginning and ending of each event in the plurality of events in the machine data, each event in the plurality of events including a portion of the machine data segmented for that event, the plurality of events including both events produced from a first data resource and events produced from a second data resource that is different from the first data resource, the machine data in one or more events produced from the first data resource having a different data format than the machine data in one or more events produced from the second data resource;
determining a time stamp for each event in the plurality of events;
performing a search on machine data included in the plurality of events;
wherein the method is performed by one or more computing devices.
2. The method as recited in claim 1 , wherein organizing the received machine data into the plurality of events comprises identifying boundaries of events within the received machine data.
3. The method as recited in claim 1 , wherein organizing the received machine data into the plurality of events further comprises:
creating rules for identifying boundaries of events within the received machine data;
storing the rules in at least one storage device;
applying the rules to the received machine data to define each event in the plurality of events.
4. The method as recited in claim 1 , wherein performing a search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that matches a particular pattern.
5. The method as recited in claim 1 , wherein performing a search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that matches a particular digital signature.
6. The method as recited in claim 1 , wherein performing a search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular token.
7. The method as recited in claim 1 , wherein performing a search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular keyword.
8. The method as recited in claim 1 , wherein performing a search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular segment.
9. The method as recited in claim 1 , wherein performing a search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular segment value.
10. The method as recited in claim 1 , wherein performing a search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular extracted entity.
11. The method as recited in claim 1 , wherein performing a search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular value for an extracted entity.
12. The method as recited in claim 1 , wherein performing a search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular semantic entity.
13. The method as recited in claim 1 , wherein performing a search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular value for a semantic entity.
14. The method as recited in claim 1 , wherein performing a search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular value for a semantic entity defined by a regular expression.
15. The method as recited in claim 1 , wherein performing a search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular value for a semantic entity defined by an extraction rule.
16. The method as recited in claim 1 , wherein performing a search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that has a particular overall structure.
17. The method as recited in claim 1 , wherein performing a search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that has a particular punctuation structure.
18. The method as recited in claim 1 , further comprising:
extracting entities from machine data in an event.
19. The method as recited in claim 1 , further comprising:
segmenting data in an event into tokens.
20. The method as recited in claim 1 , further comprising:
classifying events into event types.
21. The method as recited in claim 1 , further comprising:
dynamically constructing links between events;
wherein the links represent relationships between events.
22. The method as recited in claim 1 , further comprising:
dynamically constructing links between events;
wherein the links represent relationships between events;
constructing a path by chaining event links together;
generating statistical information based on occurrences of one or more paths.
23. The method as recited in claim 1 , wherein an event comprises at least a portion of one or more lines of data within the received machine data.
24. A non-transitory computer readable storage medium, storing software instructions, which when executed by one or more processors cause performance of:
analyzing machine data stored in at least one storage device in order to segment the machine data into a plurality of events by determining beginning and ending of each event in the plurality of events in the machine data, each event in the plurality of events including a portion of the machine data segmented for that event, the plurality of events including both events produced from a first data resource and events produced from a second data resource that is different from the first data resource, the machine data in one or more events produced from the first data resource having a different data format than the machine data in one or more events produced from the second data resource;
determining a time stamp for each event in the plurality of events;
performing a search on machine data included in the plurality of events.
25. The non-transitory computer readable storage medium as recited in claim 24 , wherein organizing the received machine data into the plurality of events comprises identifying boundaries of events within the received machine data.
26. The non-transitory computer readable storage medium as recited in claim 24 , wherein performing a search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that matches a particular pattern.
27. The non-transitory computer readable storage medium as recited in claim 24 , wherein performing a search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular segment.
28. An apparatus, comprising:
a machine data organizer subsystem, implemented at least partially in hardware, that analyzes machine data stored in at least one storage device in order to segment the machine data into a plurality of events by determining beginning and ending of each event in the plurality of events in the machine data, each event in the plurality of events including a portion of the machine data segmented for that event, the plurality of events including both events produced from a first data resource and events produced from a second data resource that is different from the first data resource, the machine data in one or more events produced from the first data resource having a different data format than the machine data in one or more events produced from the second data resource;
a time stamp subsystem, implemented at least partially in hardware, that determines a time stamp for each event in the plurality of events;
a search subsystem, implemented at least partially in hardware, that performs a search on machine data included in the plurality of events.
29. The apparatus as recited in claim 28 , wherein organizing the received machine data into the plurality of events comprises identifying boundaries of events within the received machine data.
30. The apparatus as recited in claim 28 , wherein performing a search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that matches a particular pattern.