Segmenting machine data into events
Methods and apparatus consistent with the invention provide the ability to organize and build understandings of machine data generated by a variety of information-processing environments. Machine data is a product of information-processing systems (e.g., activity logs, configuration files, messages, database records) and represents the evidence of particular events that have taken place and been recorded in raw data format. In one embodiment, machine data is turned into a machine data web by organizing machine data into events and then linking events together.
1. A method comprising:
identifying a first source signature for a first source of machine data and a second source signature for a second source of machine data;
receiving machine data;
comparing a first portion of the machine data with the first source signature and a second portion of the machine data with the second source signature;
based on comparing the first portion of the machine data with the first source signature and the second portion of the machine data with the second source signature, determining the first portion of the machine data is associated with the first source of machine data and the second portion of the machine data is associated with the second source of machine data;
based on determining the first portion of the machine data is associated with the first source of machine data, segmenting the first portion of the machine data into at least one first event, wherein segmenting the first portion of the machine data into the at least one first event comprises determining a particular starting point in the first portion of the machine data and a particular ending point in the first portion of the machine data for the at least one first event;
based on determining the second portion of the machine data is associated with the second source of machine data, segmenting the second portion of the machine data into at least one second event, wherein segmenting the second portion of the machine data into at least one second event comprises determining a particular starting point in the second portion of the machine data and a particular ending point in the second portion of the machine data for the at least one second event;
identifying, in real time, a pattern that associates the at least one first event with the at least one second event; and
providing, to a computing system, information associated with the pattern that associates the at least one first event with the at least one second event.
2. The method of claim 1 , wherein the first source signature comprises a first source label and the second source signature comprise a second source label.
3. The method of claim 1 , wherein the first source of machine data is associated with a first rule and the second source of machine data is associated with a second rule.
4. The method of claim 1 , wherein the first source of machine data is associated with a first rule and the second source of machine data is associated with a second rule, wherein segmenting the first portion of the machine data into the at least one first event is based on application of the first rule to the first portion of the machine data, wherein segmenting the second portion of the machine data into the at least one second event is based on application of the second rule to the second portion of the machine data.
5. The method of claim 1 , wherein the at least one first event includes at least a portion of the first portion of the machine data, wherein the at least one second event includes at least a portion of the second portion of the machine data, wherein the first portion of the machine data includes the first source signature and the second portion of the machine data includes the second source signature.
6. The method of claim 1 , wherein identifying the first source signature and the second source signature comprises:
generating the first source signature and the second source signature.
7. The method of claim 1 , wherein identifying the first source signature and the second source signature comprises:
obtaining a collection of machine data from a plurality of sources of machine data, the plurality of sources of machine data comprising the first source of machine data and the second source of machine data; and
generating, for each of the plurality of sources of machine data, a respective source signature.
8. The method of claim 1 , wherein identifying the first source signature and the second source signature comprises:
generating one or more hash tables.
9. The method of claim 1 , wherein identifying the first source signature and the second source signature comprises:
obtaining the first source signature and the second source signature via an application programming interface.
10. The method of claim 1 , wherein one or more of the first source signature or the second source signature are based on one or more of punctuation or a token.
11. The method of claim 1 , further comprising:
clustering the machine data based on the first source signature and the second source signature.
12. The method of claim 1 , wherein comparing the first portion of the machine data with the first source signature and the second portion of the machine data with the second source signature comprises:
performing a nearest-neighbor search.
13. The method of claim 1 , wherein comparing the first portion of the machine data with the first source signature and the second portion of the machine data with the second source signature comprises:
performing a nearest-neighbor search; and
determining a first distance between the first source signature and a third signature of the first portion of the machine data and a second distance between the second source signature and a fourth signature of the second portion of the machine data.
14. The method of claim 1 , wherein comparing the first portion of the machine data with the first source signature and the second portion of the machine data with the second source signature comprises:
performing a nearest-neighbor search; and
determining a first distance between the first source signature and a third signature of the first portion of the machine data and a second distance between the second source signature and a fourth signature of the second portion of the machine data,
wherein determining the first portion of the machine data is associated with the first source of machine data is based on the first distance and determining the second portion of the machine data is associated with the second source of machine data is based on the second distance.
15. The method of claim 1 , wherein a data format of the first portion of the machine data and a data format of the second portion of the machine data are different data formats.
16. The method of claim 1 , further comprising:
outputting the first source signature and the second source signature via an application programming interface.
17. The method of claim 1 , further comprising:
comparing a third portion of the machine data with a third source signature for a third source of machine data; and
based on comparing the third portion of the machine data with the third source signature, determining the third portion of the machine data is associated with a third source of machine data.
18. The method of claim 1 , further comprising:
comparing a third portion of the machine data with a third source signature for a third source of machine data;
based on comparing the third portion of the machine data with the third source signature, determining the third portion of the machine data is not associated with a third source of machine data; and
generating a fourth source signature for the third portion of the machine data.
19. One or more non-transitory computer-readable storage media, storing one or more sequences of instructions, which when executed by one or more processors cause the one or more processors to:
identify a first source signature for a first source of machine data and a second source signature for a second source of machine data;
receive machine data;
compare a first portion of the machine data with the first source signature and a second portion of the machine data with the second source signature;
based on comparing the first portion of the machine data with the first source signature and the second portion of the machine data with the second source signature, determine the first portion of the machine data is associated with the first source of machine data and the second portion of the machine data is associated with the second source of machine data;
based on determining the first portion of the machine data is associated with the first source of machine data, segment the first portion of the machine data into at least one first event, wherein to segment the first portion of the machine data into the at least one first event, execution of the one or more sequences of instructions by the one or more processors causes the one or more processors to determine a particular starting point in the first portion of the machine data and a particular ending point in the first portion of the machine data for the at least one first event;
based on determining the second portion of the machine data is associated with the second source of machine data, segment the second portion of the machine data into at least one second event, wherein to segment the second portion of the machine data into at least one second event, the execution of the one or more sequences of instructions by the one or more processors causes the one or more processors to determine a particular starting point in the second portion of the machine data and a particular ending point in the second portion of the machine data for the at least one second event;
identify, in real time, a pattern that associates the at least one first event with the at least one second event; and
provide, to a computing system, information associated with the pattern that associates the at least one first event with the at least one second event.
20. A system comprising:
a memory containing computer-executable instructions; and
a processing device configured to execute the computer-executable instructions to cause the system to:
identify a first source signature for a first source of machine data and a second source signature for a second source of machine data;
receive machine data;
compare a first portion of the machine data with the first source signature and a second portion of the machine data with the second source signature;
based on comparing the first portion of the machine data with the first source signature and the second portion of the machine data with the second source signature, determine the first portion of the machine data is associated with the first source of machine data and the second portion of the machine data is associated with the second source of machine data;
based on determining the first portion of the machine data is associated with the first source of machine data, segment the first portion of the machine data into at least one first event, wherein to segment the first portion of the machine data into the at least one first event, execution of the computer-executable instructions by the processing device causes the system to determine a particular starting point in the first portion of the machine data and a particular ending point in the first portion of the machine data for the at least one first event;
based on determining the second portion of the machine data is associated with the second source of machine data, segment the second portion of the machine data into at least one second event, wherein to segment the second portion of the machine data into at least one second event, the execution of the computer-executable instructions by the processing device causes the system to determine a particular starting point in the second portion of the machine data and a particular ending point in the second portion of the machine data for the at least one second event;
identify, in real time, a pattern that associates the at least one first event with the at least one second event; and
provide, to a computing system, information associated with the pattern that associates the at least one first event with the at least one second event.