Identifying system performance patterns in machine data
Methods and apparatus consistent with the invention provide the ability to organize and build understandings of machine data generated by a variety of information-processing environments. Machine data is a product of information-processing systems (e.g., activity logs, configuration files, messages, database records) and represents the evidence of particular events that have taken place and been recorded in raw data format. In one embodiment, machine data is turned into a machine data web by organizing machine data into events and then linking events together.
1. A method for improving machine data analysis, comprising:
receiving machine data from two or more components in an information technology environment, the received machine data reflecting activity in the information technology environment;
recording, in real-time, behavioral patterns among machine data from two or more different components among the two or more components in the information technology environment, the machine data among the received machine data;
learning, in real-time, new types of associative relationships between the behavioral patterns based on the co-occurrence of events within a time window bounded by a window threshold, wherein the window threshold changes based on data density or an amount of time available to complete the learning;
analyzing the associative relationships to identify one or more associative relationships that frequently occur, wherein the one or more associative relationships that frequently occur are identified as representing typical system behavior in the information technology environment;
wherein the method is performed by one or more computing devices.
2. The method as recited in claim 1 , wherein one or more other associative relationships of the associative relationships are identified as anomalous system behavior.
3. The method as recited in claim 1 , wherein machine data produced from a first component of the two or more components has a different data format than machine data produced from a second data component of the two or more components.
4. The method as recited in claim 1 , wherein learning, in real-time, new types of associative relationships further comprises:
analyzing machine data from the two or more components to identify a relationship between portions of machine data.
5. The method as recited in claim 1 , wherein learning, in real-time, new types of associative relationships further comprises:
analyzing machine data from the two or more components to identify a relationship between portions of machine data;
linking the related portions of machine data to each other.
6. The method as recited in claim 1 , further comprising:
analyzing the received machine data in order to segment the machine data into a plurality of events by determining a beginning and ending of each event in the plurality of events in the received machine data, each event in the plurality of events including a portion of the received machine data segmented for that event.
7. The method as recited in claim 1 , further comprising:
analyzing the received machine data in order to segment the machine data into a plurality of events by determining a beginning and ending of each event in the plurality of events in the received machine data, each event in the plurality of events including a portion of the received machine data segmented for that event;
associating a time stamp with each event in the plurality of events, the time stamp derived from received machine data included in that event.
8. The method as recited in claim 1 , wherein one or more other associative relationships of the associative relationships are identified as deviations from typical system behavior.
9. An apparatus for improving machine data analysis, comprising:
a machine data receiving device, implemented at least partially in hardware, that receives machine data from two or more components in an information technology environment, the received machine data reflecting activity in the information technology environment;
a real-time behavioral pattern recorder device, implemented at least partially in hardware, that records, in real-time, behavioral patterns among machine data from two or more different components among the two or more components in the information technology environment, the machine data among the received machine data;
a real-time learning device, implemented at least partially in hardware, that learns, in real-time, new types of associative relationships between the behavioral patterns based on the co-occurrence of events within a time window bounded by a window threshold, wherein the window threshold changes based on data density or an amount of time available to complete the learning;
wherein the real-time learning device analyzes the associative relationships to identify one or more associative relationships that frequently occur;
wherein the one or more associative relationships that frequently occur are identified as representing typical system behavior in the information technology environment.
10. The apparatus as recited in claim 9 , wherein one or more other associative relationships of the associative relationships are identified as anomalous system behavior.
11. The apparatus as recited in claim 9 , wherein machine data produced from a first component of the one or more components has a different data format than machine data produced from a second data component of the two or more components.
12. The apparatus as recited in claim 9 , wherein one or more other associative relationships of the associative relationships are identified as deviations from typical system behavior.
13. One or more non-transitory computer-readable storage media, storing software instructions for improving machine data analysis, which when executed by one or more processors cause performance of:
receiving machine data from two or more components in an information technology environment, the received machine data reflecting activity in the information technology environment;
recording, in real-time, behavioral patterns among machine data from two or more different components among the two or more components in the information technology environment, the machine data among the received machine data;
learning, in real-time, new types of associative relationships between the behavioral patterns based on the co-occurrence of events within a time window bounded by a window threshold, wherein the window threshold changes based on data density or an amount of time available to complete the learning;
analyzing the associative relationships to identify one or more associative relationships that frequently occur, wherein the one or more associative relationships that frequently occur are identified as representing typical system behavior in the information technology environment.
14. The one or more non-transitory computer-readable storage media as recited in claim 13 , wherein one or more other associative relationships of the associative relationships are identified as anomalous system behavior.
15. The one or more non-transitory computer-readable storage media as recited in claim 13 , wherein machine data produced from a first component of the two or more components has a different data format than machine data produced from a second data component of the two or more components.
16. The one or more non-transitory computer-readable storage media as recited in claim 13 , wherein learning, in real-time, new types of associative relationships further comprises:
analyzing machine data from the two or more components to identify a relationship between portions of machine data.
17. The one or more non-transitory computer-readable storage media as recited in claim 13 , wherein learning, in real-time, new types of associative relationships further comprises:
analyzing machine data from the two or more components to identify a relationship between portions of machine data;
linking the related portions of machine data to each other.
18. The one or more non-transitory computer-readable storage media as recited in claim 13 , wherein the instructions, when executed by the one or more computing devices, further cause performance of:
analyzing the received machine data in order to segment the machine data into a plurality of events by determining a beginning and ending of each event in the plurality of events in the received machine data, each event in the plurality of events including a portion of the received machine data segmented for that event.
19. The one or more non-transitory computer-readable storage media as recited in claim 13 , wherein the instructions, when executed by the one or more computing devices, further cause performance of:
analyzing the received machine data in order to segment the machine data into a plurality of events by determining a beginning and ending of each event in the plurality of events in the received machine data, each event in the plurality of events including a portion of the received machine data segmented for that event;
associating a time stamp with each event in the plurality of events, the time stamp derived from received machine data included in that event.
20. The one or more non-transitory computer-readable storage media as recited in claim 13 , wherein one or more other associative relationships of the associative relationships are identified as deviations from typical system behavior.