Determining system behavior using event patterns in machine data
Methods and apparatus consistent with the invention provide the ability to organize and build understandings of machine data generated by a variety of information-processing environments. Machine data is a product of information-processing systems (e.g., activity logs, configuration files, messages, database records) and represents the evidence of particular events that have taken place and been recorded in raw data format. In one embodiment, machine data is turned into a machine data web by organizing machine data into events and then linking events together.
1. A method, comprising:
receiving machine data from two or more components in an information technology environment, the machine data reflecting activity in the information technology environment;
identifying, in real-time, a first behavioral pattern between a first plurality of events from first machine data from a first component of the two or more components and a second behavioral pattern between a second plurality of events from second machine data from a second component of the two or more components;
identifying a first event from the first plurality of events from the first machine data and a second event from the second plurality of events from the second machine data;
determining a third behavioral pattern between the first event and the second event based on a co-occurrence of the first event and the second event within a time window;
identifying, in real-time, an associative relationship between the first behavioral pattern and the second behavioral pattern based on the third behavioral pattern;
determining that the associative relationship represents typical system behavior in the information technology environment; and
sending information associated with the associative relationship to a computing system;
wherein the method is performed by one or more computing devices.
2. The method as recited in claim 1 , wherein the typical system behavior identifies typical system processes.
3. The method as recited in claim 1 , wherein the typical system behavior identifies relationships between the first plurality of events from the first machine data and the second plurality of events from the second machine data.
4. The method as recited in claim 1 , wherein the typical system behavior identifies relationships between the first plurality of events from the first machine data and the second plurality of events from the second machine data, wherein the first plurality of events and the second plurality of events co-occur within the time window.
5. The method as recited in claim 1 , wherein the typical system behavior identifies that the first plurality of events and the second plurality of events are connected as part of a larger activity in the information technology environment.
6. The method as recited in claim 1 , wherein the first machine data from the first component has a different data format than the second machine data from the second component.
7. The method as recited in claim 1 , the method further comprising:
determining a number of occurrences of the third behavioral pattern, and
determining that the number of occurrences satisfies a threshold;
wherein identifying the associative relationship is based at least in part on determining that the number of occurrences satisfies the threshold.
8. The method as recited in claim 1 , wherein the identifying the first behavioral pattern and the second behavioral pattern comprises:
analyzing the first machine data from the first component to identify a first relationship between the first event and a third event from the first machine data; and
analyzing the second machine data from the second component to identify a second relationship between the second event and a fourth event from the second machine data.
9. The method as recited in claim 1 , wherein the identifying the first behavioral pattern and the second behavioral pattern comprises:
analyzing the first machine data from the first component to identify a first relationship between the first event and a third event from the first machine data;
linking the first event and the third event based at least in part on the first relationship;
analyzing the second machine data from the second component to identify a second relationship between the second event and a fourth event from the second machine data; and
linking the second event and the fourth event based at least in part on the second relationship.
10. The method as recited in claim 1 , further comprising:
analyzing the first machine data in order to segment the first machine data into the first plurality of events by determining a beginning and ending of each event in the first plurality of events from the first machine data, each event in the first plurality of events including a portion of the first machine data segmented for that event; and
analyzing the second machine data in order to segment the second machine data into the second plurality of events by determining a beginning and ending of each event in the second plurality of events from the second machine data, each event in the second plurality of events including a portion of the second machine data segmented for that event.
11. The method as recited in claim 1 , further comprising:
analyzing the first machine data in order to segment the first machine data into the first plurality of events by determining a beginning and ending of each event in the first plurality of events from the first machine data, each event in the first plurality of events including a portion of the first machine data segmented for that event;
associating a first time stamp with each event in the first plurality of events, the first time stamp derived from the portion of the first machine data segmented for that event;
analyzing the second machine data in order to segment the second machine data into the second plurality of events by determining a beginning and ending of each event in the second plurality of events from the second machine data, each event in the second plurality of events including a portion of the second machine data segmented for that event; and
associating a second time stamp with each event in the second plurality of events, the second time stamp derived from the portion of the second machine data segmented for that event.
12. One or more non-transitory computer-readable storage media, storing one or more sequences of instructions, which when executed by one or more processors cause the one or more processors to:
receive machine data from two or more components in an information technology environment, the machine data reflecting activity in the information technology environment;
identify, in real-time, a first behavioral pattern between a first plurality of events from first machine data from a first component of the two or more components and a second behavioral pattern between a second plurality of events from second machine data from a second component of the two or more components;
identify a first event from the first plurality of events from the first machine data and a second event from the second plurality of events from the second machine data;
determine a third behavioral pattern between the first event and the second event based on a co-occurrence of the first event and the second event within a time window;
identify, in real-time, an associative relationship between the first behavioral pattern and the second behavioral pattern based on the third behavioral pattern;
determine that the associative relationship represents typical system behavior in the information technology environment; and
send information associated with the associative relationship to a computing system.
13. The one or more non-transitory computer-readable storage media as recited in claim 12 , wherein the typical system behavior identifies typical system processes.
14. The one or more non-transitory computer-readable storage media as recited in claim 12 , wherein the typical system behavior identifies relationships between the first plurality of events from the first machine data and the second plurality of events from the second machine data.
15. The one or more non-transitory computer-readable storage media as recited in claim 12 , wherein the typical system behavior identifies relationships between the first plurality of events from the first machine data and the second plurality of events from the second machine data, wherein the first plurality of events and the second plurality of events co-occur within the time window.
16. The one or more non-transitory computer-readable storage media as recited in claim 12 , wherein the typical system behavior identifies that the first plurality of events and the second plurality of events are connected as part of a larger activity in the information technology environment.
17. The one or more non-transitory computer-readable storage media as recited in claim 12 , wherein execution of the one or more sequences of instructions by the one or more processors further causes the one or more processors to:
determine a number of occurrences of the third behavioral pattern, and
determine that the number of occurrences satisfies a threshold;
wherein identifying the associative relationship is based at least in part on determining that the number of occurrences satisfies the threshold.
18. The one or more non-transitory computer-readable storage media as recited in claim 12 , wherein to identify the first behavioral pattern and the second behavioral pattern, execution of the one or more sequences of instructions by the one or more processors further causes the one or more processors to:
analyze the first machine data from the first component to identify a first relationship between the first event and a third event from the first machine data; and
analyze the second machine data from the second component to identify a second relationship between the second event and a fourth event from the second machine data.
19. The one or more non-transitory computer-readable storage media as recited in claim 12 , wherein execution of the one or more sequences of instructions by the one or more processors further causes the one or more processors to:
analyze the first machine data in order to segment the first machine data into the first plurality of events by determining a beginning and ending of each event in the first plurality of events from the first machine data, each event in the first plurality of events including a portion of the first machine data segmented for that event;
associate a first time stamp with each event in the first plurality of events, the first time stamp derived from the portion of the first machine data segmented for that event;
analyze the second machine data in order to segment the second machine data into the second plurality of events by determining a beginning and ending of each event in the second plurality of events from the second machine data, each event in the second plurality of events including a portion of the second machine data segmented for that event; and
associate a second time stamp with each event in the second plurality of events, the second time stamp derived from the portion of the second machine data segmented for that event.
20. An apparatus, comprising:
one or more processors; and
a memory storing instructions, which when executed by the one or more processors, causes the one or more processors to:
receive machine data from two or more components in an information technology environment, the machine data reflecting activity in the information technology environment;
identify, in real-time, a first behavioral pattern between a first plurality of events from first machine data from a first component of the two or more components and a second behavioral pattern between a second plurality of events from second machine data from a second component of the two or more components;
identify a first event from the first plurality of events from the first machine data and a second event from the second plurality of events from the second machine data;
determine a third behavioral pattern between the first event and the second event based on a co-occurrence of the first event and the second event within a time window;
identify, in real-time, an associative relationship between the first behavioral pattern and the second behavioral pattern based on the third behavioral pattern;
determine that the associative relationship represents typical system behavior in the information technology environment; and
send information associated with the associative relationship to a computing system.