Machine data web
Methods and apparatus consistent with the invention provide the ability to organize and build understandings of machine data generated by a variety of information-processing environments. Machine data is a product of information-processing systems (e.g., activity logs, configuration files, messages, database records) and represents the evidence of particular events that have taken place and been recorded in raw data format. In one embodiment, machine data is turned into a machine data web by organizing machine data into events and then linking events together.
1. An apparatus comprising:
a memory for storing instructions; and
a processor that executes instructions to enable actions, including;
receiving first machine data from a first source;
receiving second machine data from a second source, wherein there is no explicit common structure shared among the first machine data and the second machine data;
generating, for each of the first machine data and the second machine data, a source signature that includes a result of a hash function, wherein the result of the hash function is built of a plurality of elements within the data;
determining, for each of the first machine data and the second machine data, a domain for the data using the result of the hash function;
determining, for each of the first machine data and the second machine data, a rule for identifying separate events within the machine data, wherein the rule for the first machine data is different from the rule for the second machine data, and wherein the rule is determined based on the determined domain; and
automatically organizing, for each of the first machine data and the second machine data, the received machine data into a plurality of events using the determined rule.
2. The apparatus of claim 1 , wherein the organizing, for each of the first machine data and the second machine data, comprises:
identifying event boundaries within the respective machine data; and
defining individual events as the machine data existing between adjacent boundaries.
3. The apparatus of claim 1 , wherein the result of the hash function includes an element reflecting punctuation in the machine data.
4. The apparatus of claim 1 , wherein the organizing further comprises classifying the plurality of events into one or more categories based on the source signature.
5. The apparatus of claim 1 , wherein the actions further comprise searching through the organized events.
6. The apparatus of claim 1 , wherein the actions further comprise, for each of the rule for the first machine data and the rule for the second machine data, learning the rule based on previously received machine data.
7. The apparatus of claim 1 , wherein the first and second received machine data are collected by a module, wherein one or more parts of the module are distributable to different locations having access to the machine data.
8. The apparatus of claim 1 , wherein the plurality of events are organized into one or more collections that correspond to the one or more activities for the information space.
9. The apparatus of claim 1 , wherein the actions further comprise constructing links between events in the plurality of events based on relative occurrences.
10. A computer-implemented method comprising:
receiving first machine data from a first source;
receiving second machine data from a second source, wherein there is no explicit common structure shared among the first machine data and the second machine data;
generating, for each of the first machine data and the second machine data, a source signature that includes a result of a hash function, wherein the result of the hash function is built of a plurality of elements within the data;
determining, for each of the first machine data and the second machine data, a domain for the data using the result of the hash function;
determining, for each of the first machine data and the second machine data, a rule for identifying separate events within the machine data, wherein the rule for the first machine data is different from the rule for the second machine data, and wherein the rule is determined based on the determined domain; and
automatically organizing, for each of the first machine data and the second machine data, the received machine data into a plurality of events using the determined rule.
11. The method of claim 10 , wherein the organizing, for each of the first machine data and the second machine data, comprises:
identifying event boundaries within the respective machine data; and
defining individual events as the machine data existing between adjacent boundaries.
12. The method of claim 10 , wherein the result of the hash function includes an element reflecting punctuation in the machine data.
13. The method of claim 10 , wherein the organizing further comprises classifying the plurality of events into one or more categories based on the source signature.
14. The method of claim 10 , further comprising searching through the organized events.
15. The method of claim 10 , further comprising, for each of the rule for the first machine data and the rule for the second machine data, learning the rule based on previously received machine data.
16. The method of claim 10 , wherein the first and second received machine data are collected by a module, wherein one or more parts of the module are distributable to different locations having access to the machine data.
17. The method of claim 10 , wherein the plurality of events are organized into one or more collections that correspond to the one or more activities for the information space.
18. The method of claim 10 , further comprising constructing links between events in the plurality of events based on relative occurrences.
19. A processor readable non-transitory storage media that stores executable instructions, wherein the execution of the instructions by a processor enables actions, the actions comprising:
receiving first machine data from a first source;
receiving second machine data from a second source, wherein there is no explicit common structure shared among the first machine data and the second machine data;
generating, for each of the first machine data and the second machine data, a result of a hash function, wherein the result of the hash function is built of a plurality of elements within the data;
determining, for each of the first machine data and the second machine data, a domain for the data using the result of the hash function;
determining, for each of the first machine data and the second machine data, a rule for identifying separate events within the machine data, wherein the rule for the first machine data is different from the rule for the second machine data, and wherein the rule is determined based on the determined domain; and
automatically organizing, for each of the first machine data and the second machine data, the received machine data into a plurality of events using the determined rule.
20. The media of claim 19 , wherein the organizing, for each of the first machine data and the second machine data, comprises:
identifying event boundaries within the respective machine data; and
defining individual events as the machine data existing between adjacent boundaries.
21. The media of claim 19 , wherein the result of the hash function includes an element reflecting punctuation in the machine data.
22. The media of claim 19 , wherein the organizing further comprises classifying the plurality of events into one or more categories based on the source signature.
23. The media of claim 19 , wherein the actions further comprise searching through the organized events.
24. The media of claim 19 , wherein the actions further comprise, for each of the rule for the first machine data and the rule for the second machine data, learning the rule based on previously received machine data.
25. The media of claim 19 , wherein the first and second received machine data are collected by a module, wherein one or more parts of the module are distributable to different locations having access to the machine data.
26. The media of claim 19 , wherein the plurality of events are organized into one or more collections that correspond to the one or more activities for the information space.
27. The media of claim 19 , wherein the actions further comprise constructing links between events in the plurality of events based on relative occurrences.