Uniform storage and search of security-related events derived from machine data from different sources
Methods and apparatus consistent with the invention provide the ability to organize and build understandings of machine data generated by a variety of information-processing environments. Machine data is a product of information-processing systems (e.g., activity logs, configuration files, messages, database records) and represents the evidence of particular events that have taken place and been recorded in raw data format. In one embodiment, machine data is turned into a machine data web by organizing machine data into events and then linking events together.
1. A method, comprising:
analyzing machine data stored in at least one storage device in order to segment the machine data into a plurality of events by identifying a source of the machine data, selecting a rule from a plurality of rules based on the identified source, the rule identifies event boundaries in machine data from the identified source, and segmenting the machine data into the plurality of events by determining a beginning and ending of each event in the plurality of events in the machine data using the rule, the machine data related to security aspects of one or more information technology systems, each event in the plurality of events including a portion of the machine data segmented for that event;
determining a time stamp for each event in the plurality of events from the portion of machine data included in each event;
examining values in the portions of machine data included in events in the plurality of events using extraction rules to extract the values from the portions of machine data;
wherein the method is performed by one or more computing devices.
2. The method as recited in claim 1 , wherein determining the beginning and ending of each event comprises identifying boundaries of events within the received machine data.
3. The method as recited in claim 1 , wherein determining the beginning and ending of each event further comprises:
creating rules for identifying boundaries of events within the machine data;
storing the rules in at least one storage device;
applying the rules to the machine data to define each event in the plurality of events.
4. The method as recited in claim 1 , wherein performing the search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that matches a particular pattern.
5. The method as recited in claim 1 , wherein performing the search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that matches a particular digital signature.
6. The method as recited in claim 1 , wherein performing the search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular token.
7. The method as recited in claim 1 , wherein performing the search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular keyword.
8. The method as recited in claim 1 , wherein performing the search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular segment.
9. The method as recited in claim 1 , wherein performing the search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular segment value.
10. The method as recited in claim 1 , wherein performing the search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular extracted entity.
11. The method as recited in claim 1 , wherein performing the search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular value for an extracted entity.
12. The method as recited in claim 1 , wherein performing the search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular semantic entity.
13. The method as recited in claim 1 , wherein performing the search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular value for a semantic entity.
14. The method as recited in claim 1 , wherein performing the search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular value for a semantic entity defined by a regular expression.
15. The method as recited in claim 1 , wherein performing the search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular value for a semantic entity defined by an extraction rule.
16. The method as recited in claim 1 , wherein performing the search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that has a particular overall structure.
17. The method as recited in claim 1 , wherein performing the search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that has a particular punctuation structure.
18. The method as recited in claim 1 , further comprising:
extracting entities from machine data in an event.
19. The method as recited in claim 1 , further comprising:
segmenting data in an event into tokens.
20. The method as recited in claim 1 , further comprising:
classifying events into event types.
21. The method as recited in claim 1 , further comprising:
dynamically constructing links between events;
wherein the links represent relationships between events.
22. The method as recited in claim 1 , further comprising:
dynamically constructing links between events;
wherein the links represent relationships between events;
constructing a path by chaining event links together;
generating statistical information based on occurrences of one or more paths.
23. The method as recited in claim 1 , wherein an event comprises at least a portion of one or more lines of data within the received machine data.
24. One or more non-transitory computer-readable storage media, storing one or more sequences of instructions, which when executed by one or more processors cause performance of:
analyzing machine data stored in at least one storage device in order to segment the machine data into a plurality of events by identifying a source of the machine data, selecting a rule from a plurality of rules based on the identified source, the rule identifies event boundaries in machine data from the identified source, and segmenting the machine data into the plurality of events by determining a beginning and ending of each event in the plurality of events in the machine data using the rule, the machine data related to security aspects of one or more information technology systems, each event in the plurality of events including a portion of the machine data segmented for that event;
determining a time stamp for each event in the plurality of events from the portion of machine data included in each event;
examining values in the portions of machine data included in events in the plurality of events using extraction rules to extract the values from the portions of machine data.
25. The one or more non-transitory computer-readable storage media as recited in claim 24 , wherein determining the beginning and ending of each event comprises identifying boundaries of events within the received machine data.
26. The one or more non-transitory computer-readable storage media as recited in claim 24 , wherein performing the search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that matches a particular pattern.
27. The one or more non-transitory computer-readable storage media as recited in claim 24 , wherein performing the search on the plurality of events comprises identifying an event in the plurality of events that includes machine data that includes a particular segment.
28. An apparatus, comprising:
a machine data organizer subsystem, implemented at least partially in hardware, that analyzes machine data stored in at least one storage device in order to segment the machine data into a plurality of events by identifying a source of the machine data, selecting a rule from a plurality of rules based on the identified source, the rule identifies event boundaries in machine data from the identified source, and segmenting the machine data into the plurality of events by determining a beginning and ending of each event in the plurality of events in the machine data using the rule, the machine data related to security aspects of one or more information technology systems, each event in the plurality of events including a portion of the machine data segmented for that event;
a time stamp subsystem, implemented at least partially in hardware, that determines a time stamp for each event in the plurality of events from the portion of machine data included in each event;
a search subsystem, implemented at least partially in hardware, that examines values in the portions of machine data included in events in the plurality of events using extraction rules to extract the values from the portions of machine data.
29. The apparatus as recited in claim 28 , wherein the machine data organizer subsystem identifies boundaries of events within the received machine data.
30. The apparatus as recited in claim 28 , wherein search subsystem identifies an event in the plurality of events that includes machine data that matches a particular pattern.