Customized reporting and mining of event data
Event data (e.g., log messages) are represented as sets of attribute/value pairs. An index maps each attribute/value pair or attribute/value tuple to a pointer that points to event data which contains the attribute/value pair or attribute/value tuple. An attribute co-occurrence map or matrix can be generated that includes attribute names that co-occur together. Queries and custom reports can be generated by projecting event data into one or more attributes or attribute/value pairs, and then determining statistics on other attributes using a combination of the inverted index, the attribute co-occurrence map or matrix, operations on sets and/or math and statistical functions.
1. A computer-implemented method, comprising:
receiving event data in a log file;
transforming the event data into attribute/value pairs;
generating an index mapping at least one of the attribute/value pairs to a reference pointer that references an instance of event data in the log file, the instance of event data in the log file including a textual message from which the attribute/value pair was transformed;
generating an attribute co-occurrence data structure, the attribute co-occurrence data structure identifying a relationship between a first attribute in the attribute/value pairs and a second attribute in the attribute/value pairs, where existence of the relationship between the first attribute and the second attribute indicates that reports for finding a textual message including both the first attribute and the second attribute are allowable; and
in response to a query including the first attribute, generating a report based on the attribute co-occurrence data structure and the attribute/value pairs in the index, including:
identifying the second attribute based on the relationship identified in the attribute co-occurrence data structure;
identifying an attribute/value pair that includes the second attribute; and
generating the report, including providing access to the event data in the log file using a reference pointer that is mapped to the identified attribute/value pair in the index,
where the method is performed by one or more processors.
2. The method of claim 1 , where identifying the relationship between the first attribute and the second attribute includes identifying a sender-recipient relationship between the first attribute and the second attribute according to an email message.
3. The method of claim 1 , wherein generating the report further comprises:
projecting the event data into one or more attributes or attribute/value pairs; and
determining statistics on other attributes using a combination of the index and the attribute co-occurrence data structure.
4. The method of claim 3 , wherein the statistics are determined using operations on sets of attribute/value pairs.
5. The method of claim 3 , wherein the statistics are determined using statistical functions.
6. The method of claim 1 , wherein the index maps an attribute/value tuple to a reference pointer that references an instance of event data that contains the attribute/value tuple.
7. The method of claim 6 , further comprising:
determining a count representing a number of occurrences of an attribute/value pair or attribute/value tuple for a predetermined period of time.
8. A system, comprising:
a storage device operable for storing event data in a log file; and
one or more data processing devices configured to perform operations comprising:
transforming the event data into attribute/value pairs;
generating an index mapping at least one of the attribute/value pairs to a reference pointer that references an instance of event data in the log file, the instance of event data in the log file including a textual message from which the attribute/value pair was transformed;
generating an attribute co-occurrence data structure, the attribute co-occurrence data structure identifying a relationship between a first attribute in the attribute/value pairs and a second attribute in the attribute/value pairs, where existence of the relationship between the first attribute and the second attribute indicates that reports for finding a textual message including both the first attribute and the second attribute are allowable; and
in response to a query including the first attribute, generating a report based on the attribute co-occurrence data structure and the attribute/value pairs in the index, including:
identifying the second attribute based on the relationship identified in the attribute co-occurrence data structure;
identifying an attribute/value pair that includes the second attribute; and
generating the report, including providing access to the event data in the log file using a reference pointer that is mapped to the identified attribute/value pair in the index.
9. The system of claim 8 , wherein the event data includes log messages.
10. The system of claim 8 , wherein the index is operable to determine a count representing a number of occurrences of an attribute/value pair or attribute/value tuple for a predetermined period of time.
11. The system of claim 8 , further comprising:
a pre-parser adapted for coupling with a number of collectors to receive the event data, the pre-parser operable for identifying a source of the event data.
12. The system of claim 8 , wherein the storage device is a persistent storage device configured to store the event data for a user-specified period of time.
13. The system of claim 8 , wherein transforming the event data into attribute/value pairs includes generating the attribute/value pairs by applying rules to the event data.
14. The system of claim 13 , wherein the rules include a first set of rules for generating the attribute/value pairs from the event data, and a second set of rules for segmenting a space of the attribute/value pairs into regions and labeling the regions with new attribute/value pairs.
15. The system of claim 8 , wherein the attribute/value pairs represent particular instances of event data.
16. The system of claim 8 , wherein at least one attribute in an attribute/value pair is an action result.
17. The system of claim 8 , wherein at least one attribute in an attribute/value pair is a user name.
18. The system of claim 8 , wherein at least one attribute is user-definable.
19. The system of claim 8 , wherein the attribute co-occurrence data structure comprises a two-dimensional matrix having rows and columns, each row and column representing a different attribute, the matrix including a value at each intersection of a row and column of the matrix, the value for indicating a relationship between the attributes represented by the intersecting row and column.