Machine data web
Methods and apparatus consistent with the invention provide the ability to organize and build understandings of machine data generated by a variety of information-processing environments. Machine data is a product of information-processing systems (e.g., activity logs, configuration files, messages, database records) and represents the evidence of particular events that have taken place and been recorded in raw data format. In one embodiment, machine data is turned into a machine data web by organizing machine data into events and then linking events together.
1. A method, comprising:
organizing into a plurality of events machine data from a plurality of diverse sources, wherein there is no obvious common structure shared among the machine data from the plurality of diverse sources, and wherein machine data included in each event in the plurality of events retains its original structure;
extracting IP addresses from the plurality of events;
performing a search on the plurality of events;
wherein the method is performed by one or more computing devices.
2. The method as recited in claim 1 , further comprising:
determining a time stamp for each event in the plurality of events.
3. The method as recited in claim 1 , further comprising:
automatically identifying boundaries of events within the machine data.
4. The method as recited in claim 1 , further comprising:
generating statistical information based on extracted event information.
5. The method as recited in claim 1 , wherein extracting IP addresses further comprises:
applying regular expressions to the plurality of events to extract the IP addresses from the plurality of events.
6. The method as recited in claim 1 , wherein extracting IP addresses further comprises:
applying search trees to the plurality of events to extract the IP addresses from the plurality of events.
7. The method as recited in claim 1 , further comprising:
automatically identifying boundaries of events within the machine data.
8. The method as recited in claim 1 , wherein organizing machine data into a plurality of events further comprises:
automatically creating rules for identifying boundaries of events within the machine data;
storing the rules in at least one storage device;
applying the rules to the machine data to define each event in the plurality of events.
9. The method as recited in claim 1 , wherein an event is comprised of at least a portion of one or more lines of data within the machine data.
10. An apparatus, comprising:
a machine data organization subsystem, implemented at least partially in hardware, that organizes into a plurality of events machine data from a plurality of diverse sources, wherein there is no obvious common structure shared among the machine data from the plurality of diverse sources, and wherein machine data included in each event in the plurality of events retains its original structure;
an IP extraction subsystem, implemented at least partially in hardware, that extracts IP addresses from the plurality of events;
an event search subsystem, implemented at least partially in hardware, that performs a search on the plurality of events.
11. The apparatus as recited in claim 10 , further comprising:
a time stamp subsystem, implemented at least partially in hardware, that determines a time stamp for each event in the plurality of events.
12. The apparatus as recited in claim 10 , further comprising:
a boundary identification subsystem, implemented at least partially in hardware, that automatically identifies boundaries of events within the machine data.
13. The apparatus as recited in claim 10 , further comprising:
a statistical generation subsystem, implemented at least partially in hardware, that generates statistical information based on extracted event information.
14. The apparatus as recited in claim 10 , wherein the IP extraction subsystem further comprises:
a subsystem, implemented at least partially in hardware, that applies regular expressions to the plurality of events to extract the IP addresses from the plurality of events.
15. The apparatus as recited in claim 10 , wherein the IP extraction subsystem further comprises:
a subsystem, implemented at least partially in hardware, that applies search trees to the plurality of events to extract the IP addresses from the plurality of events.
16. The apparatus as recited in claim 10 , further comprising:
a boundary identification subsystem, implemented at least partially in hardware, that automatically identifies boundaries of events within the machine data.
17. The apparatus as recited in claim 10 , wherein the machine data organization organizing subsystem further comprises:
a subsystem, implemented at least partially in hardware, that automatically creates rules for identifying boundaries of events within the machine data;
a subsystem, implemented at least partially in hardware, that stores the rules in at least one storage device;
a subsystem, implemented at least partially in hardware, that applies the rules to the machine data to define each event in the plurality of events.
18. The apparatus as recited in claim 10 , wherein an event is comprised of at least a portion of one or more lines of data within the machine data.
19. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to perform:
organizing into a plurality of events machine data from a plurality of diverse sources, wherein there is no obvious common structure shared among the machine data from the plurality of diverse sources, and wherein machine data included in each event in the plurality of events retains its original structure;
extracting IP addresses from the plurality of events;
performing a search on the plurality of events.
20. The non-transitory computer-readable medium as recited in claim 19 , wherein execution of the one or more sequences of instructions by the one or more processors causes the one or more processors to further perform:
determining a time stamp for each event in the plurality of events.
21. The non-transitory computer-readable medium as recited in claim 19 , wherein execution of the one or more sequences of instructions by the one or more processors causes the one or more processors to further perform:
automatically identifying boundaries of events within the machine data.
22. The non-transitory computer-readable medium as recited in claim 19 , wherein execution of the one or more sequences of instructions by the one or more processors causes the one or more processors to further perform:
generating statistical information based on extracted event information.
23. The non-transitory computer-readable medium as recited in claim 19 , wherein extracting IP addresses further comprises:
applying regular expressions to the plurality of events to extract the IP addresses from the plurality of events.
24. The non-transitory computer-readable medium as recited in claim 19 , wherein extracting IP addresses further comprises:
applying search trees to the plurality of events to extract the IP addresses from the plurality of events.
25. The non-transitory computer-readable medium as recited in claim 19 , wherein execution of the one or more sequences of instructions by the one or more processors causes the one or more processors to further perform:
automatically identifying boundaries of events within the machine data.
26. The non-transitory computer-readable medium as recited in claim 19 , wherein organizing machine data into a plurality of events further comprises:
automatically creating rules for identifying boundaries of events within the machine data;
storing the rules in at least one storage device;
applying the rules to the machine data to define each event in the plurality of events.
27. The non-transitory computer-readable medium as recited in claim 19 , wherein an event is comprised of at least a portion of one or more lines of data within the machine data.