Searching of events derived from machine data using field and keyword criteria
Methods and apparatus consistent with the invention provide the ability to organize and build understandings of machine data generated by a variety of information-processing environments. Machine data is a product of information-processing systems (e.g., activity logs, configuration files, messages, database records) and represents the evidence of particular events that have taken place and been recorded in raw data format. In one embodiment, machine data is turned into a machine data web by organizing machine data into events and then linking events together.
1. A method, comprising:
analyzing machine data stored in at least one storage device in order to segment the machine data into a plurality of events by determining beginning and ending of each event in the plurality of events in the machine data, each event in the plurality of events including a portion of the machine data segmented for that event, the plurality of events including both events produced from a first data resource and events produced from a second data resource that is different from the first data resource, the machine data in one or more events produced from the first data resource having a different data format than the machine data in one or more events produced from the second data resource;
associating a time stamp with each event in the plurality of events, the time stamp derived from machine data included in that event;
performing a search on the plurality of events by identifying in the plurality of events at least one event having machine data that matches a pattern;
wherein the method is performed by one or more computing devices.
2. The method as recited in claim 1 , wherein the pattern for the search is based on having a particular token in a portion of machine data in an event.
3. The method as recited in claim 1 , wherein the pattern for the search is based on having a particular segment in a portion of machine data in an event.
4. The method as recited in claim 1 , wherein the pattern for the search is based on having a particular segment value in a portion of machine data in an event.
5. The method as recited in claim 1 , wherein the pattern for the search is based on having a particular extracted entity in a portion of machine data in an event.
6. The method as recited in claim 1 , wherein the pattern for the search is based on having in a portion of machine data in an event a particular value for an extracted entity.
7. The method as recited in claim 1 , wherein the pattern for the search is based on having a particular semantic entity in a portion of machine data in an event.
8. The method as recited in claim 1 , wherein the pattern for the search is based on having in a portion of machine data in an event a particular value for a particular semantic entity.
9. The method as recited in claim 1 , wherein the pattern for the search is based on having in a portion of machine data in an event a particular value for a particular semantic entity defined by a regular expression.
10. The method as recited in claim 1 , wherein the pattern for the search is based on having in a portion of machine data in an event a particular value for a particular semantic entity defined by an extraction rule.
11. The method as recited in claim 1 , wherein the pattern for the search is based on having in a portion of machine data in an event both (a) a particular token or keyword, and (b) a particular value for a particular semantic entity defined by an extraction rule or regular expression.
12. The method as recited in claim 1 , wherein the pattern for the search is based on an overall structure of the portion of machine data in an event.
13. The method as recited in claim 1 , wherein the pattern for the search is based on having a particular punctuation structure in a portion of machine data in an event.
14. The method as recited in claim 1 , wherein the pattern for the search is based on having both a particular punctuation structure and one or more particular tokens in a portion of machine data in an event.
15. The method as recited in claim 1 , wherein the pattern for the search includes a digital signature.
16. The method as recited in claim 1 , further comprising receiving a search query specifying the pattern to be used in performing the search.
17. A non-transitory computer readable storage medium, storing software instructions, which when executed by one or more processors cause performance of:
analyzing machine data stored in at least one storage device in order to segment the machine data into a plurality of events by determining beginning and ending of each event in the plurality of events in the machine data, each event in the plurality of events including a portion of the machine data segmented for that event, the plurality of events including both events produced from a first data resource and events produced from a second data resource that is different from the first data resource, the machine data in one or more events produced from the first data resource having a different data format than the machine data in one or more events produced from the second data resource;
associating a time stamp with each event in the plurality of events, the time stamp derived from machine data included in that event;
performing a search on the plurality of events by identifying in the plurality of events at least one event having machine data that matches a pattern.
18. The non-transitory computer readable storage medium as recited in claim 17 , wherein the pattern for the search is based on having a particular token in a portion of machine data in an event.
19. The non-transitory computer readable storage medium as recited in claim 17 , wherein the pattern for the search is based on having a particular segment in a portion of machine data in an event.
20. The non-transitory computer readable storage medium as recited in claim 17 , wherein the pattern for the search is based on having a particular segment value in a portion of machine data in an event.
21. An apparatus, comprising:
a machine data organizer subsystem, implemented at least partially in hardware, that analyzes machine data stored in at least one storage device in order to segment the machine data into a plurality of events by determining beginning and ending of each event in the plurality of events in the machine data, each event in the plurality of events including a portion of the machine data segmented for that event, the plurality of events including both events produced from a first data resource and events produced from a second data resource that is different from the first data resource, the machine data in one or more events produced from the first data resource having a different data format than the machine data in one or more events produced from the second data resource;
a time stamp subsystem, implemented at least partially in hardware, that associates a time stamp with each event in the plurality of events, the time stamp derived from machine data included in that event;
a search subsystem, implemented at least partially in hardware, that performs a search on the plurality of events by identifying in the plurality of events at least one event having machine data that matches a pattern.
22. The apparatus as recited in claim 21 , wherein the pattern for the search is based on having a particular token in a portion of machine data in an event.
23. The apparatus as recited in claim 21 , wherein the pattern for the search is based on having a particular segment in a portion of machine data in an event.
24. The apparatus as recited in claim 21 , wherein the pattern for the search is based on having a particular segment value in a portion of machine data in an event.
25. A method, comprising:
analyzing machine data stored in at least one storage device in order to segment the machine data into a plurality of events by determining beginning and ending of each event in the plurality of events in the machine data, each event in the plurality of events including a portion of the machine data segmented for that event, the plurality of events including a first set of one or more events and a second set of one or more events, the plurality of events including both events produced from a first data resource and events produced from a second data resource that is different from the first data resource, the machine data in one or more events produced from the first data resource having a different data format than the machine data in one or more events produced from the second data resource;
deriving an alphanumeric pattern from at least a portion of machine data in the first set of one or more events;
identifying, in the second set of one or more events, one or more events that include at least a portion of machine data that matches the pattern;
wherein the method is performed by one or more computing devices.
26. The method as recited in claim 25 , wherein the pattern includes a digital signature.
27. The method as recited in claim 25 , wherein the pattern is based on having one or more tokens in a portion of machine data in at least one event.
28. The method as recited in claim 25 , wherein the pattern is based on having one or more keywords in a portion of machine data in at least one event.
29. The method as recited in claim 25 , wherein the pattern is based on having one or more segments in a portion of machine data in at least one event.
30. The method as recited in claim 25 , wherein the pattern is based on having one or more segment values in a portion of machine data in at least one event.
31. The method as recited in claim 25 , wherein the pattern is based on having one or more extracted entities in a portion of machine data in at least one event.
32. The method as recited in claim 25 , wherein the pattern is based on having in a portion of machine data in at least one event a particular value for an extracted entity.
33. The method as recited in claim 25 , wherein the pattern is based on having one or more semantic entities in a portion of machine data in at least one event.
34. The method as recited in claim 25 , wherein the pattern is based on having in a portion of machine data in at least one event a particular value for a semantic entity.
35. The method as recited in claim 25 , wherein the pattern is based on having in a portion of machine data in at least one event a particular value for a semantic entity defined by a regular expression.
36. The method as recited in claim 25 , wherein the pattern is based on having in a portion of machine data in at least one event a particular value for a semantic entity defined by an extraction rule.
37. The method as recited in claim 25 , wherein the pattern is based on an overall structure of a portion of machine data in at least one event.
38. The method as recited in claim 25 , wherein the pattern is based on having a particular punctuation structure in a portion of machine data in at least one event.
39. The method as recited in claim 25 , wherein the pattern is based on having both a particular punctuation structure and one or more particular tokens in a portion of machine data in at least one event.
40. The method as recited in claim 25 , wherein the pattern is associated with an event type.
41. The method as recited in claim 25 , further comprising:
wherein the pattern is associated with an event type; and
generating statistical information for the event type.
42. The method as recited in claim 25 , further comprising:
wherein the pattern is associated with an event type;
generating statistical information for the event type; and
wherein the statistical information is accessible via an application programming interface.
43. The method as recited in claim 25 , wherein identifying, in the second set of events, one or more events whose machine data matches the pattern comprises matching the pattern using a nearest-neighbor evaluation.
44. The method as recited in claim 25 , further comprising:
segmenting the portion of machine data in an event into one or more tokens.
45. The method as recited in claim 25 , further comprising:
extracting an entity from the portion of machine data in an event.
46. The method as recited in claim 25 , further comprising:
identifying a machine data source for at least a portion of the machine data.
47. The method as recited in claim 25 , further comprising:
identifying a machine data source using at least a portion of the machine data.
48. The method as recited in claim 25 , further comprising:
constructing links between events in the plurality of events;
wherein the links represent relationships between events in the plurality of events.
49. The method as recited in claim 25 , further comprising:
constructing links between events in the plurality of events;
wherein the links represent relationships between events in the plurality of events;
constructing a path by chaining event links together;
generating statistical information based on occurrences of one or more paths.
50. The method as recited in claim 25 , wherein the portion of machine data included in an event includes one line of machine data.
51. The method as recited in claim 25 , wherein the portion of machine data included in an event includes two or more lines of machine data.
52. The method as recited in claim 25 , further comprising associating a time stamp with each event in the plurality of events.