Identification of systems with anomalous behaviour using events derived from machine data produced by those systems
Methods and apparatus consistent with the invention provide the ability to organize and build understandings of machine data generated by a variety of information-processing environments. Machine data is a product of information-processing systems (e.g., activity logs, configuration files, messages, database records) and represents the evidence of particular events that have taken place and been recorded in raw data format. In one embodiment, machine data is turned into a machine data web by organizing machine data into events and then linking events together.
1. A method for improving machine data analysis, comprising:
receiving machine data from two or more components in an Information Technology environment, the machine data including both machine data produced from a first component of the two or more components and machine data produced from a second component of the two or more components that is different from the first component, machine data produced from the first component and machine data produced from the second component reflect activity across the Information Technology environment;
organizing the machine data into a plurality of events by identifying a source of the machine data, selecting a rule from a plurality of rules based on the identified source, the rule identifies event boundaries in machine data from the identified source, and determining a beginning and ending of each event in the plurality of events in the machine data using the rule, each event in the plurality of events including a portion of the machine data segmented for that event;
identifying, by a processor, one or more patterns derived from at least a portion of machine data in events in the plurality of events by dynamically learning and identifying new types of relationships as they occur between one or more events or event types produced by the first component and one or more events or event types produced by the second component, wherein pairs of events or event types that have an identified relationship are stored and analyzed by the processor over time in order to derive a pattern among the one or more patterns;
determining whether the one or more patterns indicate anomalous behavior in the Information Technology environment, thereby allowing users to find and fix problems in the Information Technology environment.
2. The method as recited in claim 1 , wherein determining whether the one or more patterns indicate anomalous behavior further comprises:
classifying the one or more patterns as frequently occurring or infrequently occurring.
3. The method as recited in claim 1 , wherein determining whether the one or more patterns indicate anomalous behavior further comprises:
classifying the one or more patterns as frequently occurring or infrequently occurring;
wherein infrequently occurring patterns indicate anomalous behavior.
4. The method as recited in claim 1 , wherein the machine data is organized into the plurality of events as the machine data is received from the one or more systems.
5. The method as recited in claim 1 , wherein the machine data is organized into the plurality of events after the machine data is received from the one or more systems.
6. The method as recited in claim 1 , wherein the one or more patterns are identified as the machine data is being organized.
7. The method as recited in claim 1 , wherein the one or more patterns are identified after the machine data is organized.
8. The method as recited in claim 1 , further comprising associating a time stamp with each event in the plurality of events.
9. A non-transitory computer readable storage medium, storing software instructions for improving machine data analysis, which when executed by one or more processors cause performance of:
receiving machine data from two or more components in an Information Technology environment, the machine data including both machine data produced from a first component of the two or more components and machine data produced from a second component of the two or more components that is different from the first component, machine data produced from the first component and machine data produced from the second component reflect activity across the Information Technology environment;
organizing the machine data into a plurality of events by identifying a source of the machine data, selecting a rule from a plurality of rules based on the identified source, the rule identifies event boundaries in machine data from the identified source, and determining a beginning and ending of each event in the plurality of events in the machine data using the rule, each event in the plurality of events including a portion of the machine data segmented for that event;
identifying, by a processor, one or more patterns derived from at least a portion of machine data in events in the plurality of events by dynamically learning and identifying new types of relationships as they occur between one or more events or event types produced by the first component and one or more events or event types produced by the second component, wherein pairs of events or event types that have an identified relationship are stored and analyzed by the processor over time in order to derive a pattern among the one or more patterns;
determining whether the one or more patterns indicate anomalous behavior in the Information Technology environment, thereby allowing users to find and fix problems in the Information Technology environment.
10. The non-transitory computer readable storage medium as recited in claim 9 , wherein determining whether the one or more patterns indicate anomalous behavior further comprises:
classifying the one or more patterns as frequently occurring or infrequently occurring.
11. The non-transitory computer readable storage medium as recited in claim 9 , wherein determining whether the one or more patterns indicate anomalous behavior further comprises:
classifying the one or more patterns as frequently occurring or infrequently occurring;
wherein infrequently occurring patterns indicate anomalous behavior.
12. The non-transitory computer readable storage medium as recited in claim 9 , wherein the machine data is organized into the plurality of events as the machine data is received from the one or more systems.
13. The non-transitory computer readable storage medium as recited in claim 9 , wherein the machine data is organized into the plurality of events after the machine data is received from the one or more systems.
14. The non-transitory computer readable storage medium as recited in claim 9 , wherein the one or more patterns are identified as the machine data is being organized.
15. The non-transitory computer readable storage medium as recited in claim 9 , wherein the one or more patterns are identified after the machine data is organized.
16. The non-transitory computer readable storage medium as recited in claim 9 , further comprising associating a time stamp with each event in the plurality of events.
17. An apparatus for improving machine data analysis, comprising:
a machine data receiver subsystem, implemented at least partially in hardware, that receives machine data from two or more components in an Information Technology environment, the machine data including both machine data produced from a first component of the two or more components and machine data produced from a second component of the two or more components that is different from the first component, machine data produced from the first component and machine data produced from the second component reflect activity across the Information Technology environment;
a machine data organizer subsystem, implemented at least partially in hardware, that organizes the machine data into a plurality of events by identifying a source of the machine data, selecting a rule from a plurality of rules based on the identified source, the rule identifies event boundaries in machine data from the identified source, and determining a beginning and ending of each event in the plurality of events in the machine data using the rule, each event in the plurality of events including a portion of the machine data segmented for that event;
a pattern identifier subsystem, implemented at least partially in hardware, that identifies one or more patterns derived from at least a portion of machine data in events in the plurality of events by dynamically learning and identifying new types of relationships as they occur between one or more events or event types produced by the first component and one or more events or event types produced by the second component, wherein pairs of events or event types that have an identified relationship are stored and analyzed by the processor over time in order to derive a pattern among the one or more patterns;
a determination subsystem, implemented at least partially in hardware, that determines whether the one or more patterns indicate anomalous behavior in the Information Technology environment, thereby allowing users to find and fix problems in the Information Technology environment.
18. The apparatus as recited in claim 17 , wherein the determining subsystem further comprises:
a subsystem, implemented at least partially in hardware, that classifies the one or more patterns as frequently occurring or infrequently occurring.
19. The apparatus as recited in claim 17 , wherein the determining subsystem further comprises:
a subsystem, implemented at least partially in hardware, that classifies the one or more patterns as frequently occurring or infrequently occurring;
wherein infrequently occurring patterns indicate anomalous behavior.
20. The apparatus as recited in claim 17 , wherein the machine data is organized into the plurality of events as the machine data is received from the one or more systems.
21. The apparatus as recited in claim 17 , wherein the machine data is organized into the plurality of events after the machine data is received from the one or more systems.
22. The apparatus as recited in claim 17 , wherein the one or more patterns are identified as the machine data is being organized.
23. The apparatus as recited in claim 17 , wherein the one or more patterns are identified after the machine data is organized.
24. The apparatus as recited in claim 17 , further comprising associating a time stamp with each event in the plurality of events.