Anomaly detection
Analyzing log data, such as security log data and machine data, is disclosed. A baseline is built for a set of machine data. The baseline is built at least in part by determining a plurality of signature profiles for a plurality of respective time slices. An occurrence of an anomaly associated with the source of the machine data is determined. The occurrence is determined at least in part by determining that received machine data does not conform to the baseline within a threshold.
1. A computer-implemented method comprising:
identifying a first plurality of log entries received over a predefined period of time;
categorizing each log entry from the first plurality of log entries, into a category from a plurality of categories; based on a signature of the log entry, wherein the signature is based on a printf statement configured to generate log entries associated with the signature;
creating a baseline based on a number of log entries in each category, the baseline being a representative quantity received over the predefined period of time;
identifying a second plurality of log entries;
categorizing each log entry from the second plurality of log entries;
detecting an anomaly in one category from the plurality of categories based on a count of log entries from the second plurality of log entries in the category deviating from the count of log entries from the first plurality of log entries in the category by a predetermined threshold; and
causing presentation on a user interface (UI) of the detected anomaly; wherein causing presentation on a user interface (UI) of the detected anomaly further comprises:
presenting in the UI log entries with a signature not present in the baseline with a label indicating the log entries that are new; and
presenting logs missing from the baseline with a label of gone.
2. The method as recited in claim 1 , wherein the signature is associated with at least one static word in the log entry or at least one non-static portion corresponding to wild cards for words in the log entry, the method further comprising:
providing an option in the UI to edit the signature by adding wild cards in the signature.
3. The method as recited in claim 1 , further comprising:
causing presentation in the UI of a histogram based on the count of log entries, from the second plurality of log entries, for each unit of time granularity.
4. The method as recited in claim 1 , further comprising:
providing options in the UI for receiving information about the detected anomaly, the information comprising a description of the anomaly and a severity of the anomaly.
5. The method as recited in claim 1 , wherein the log entries with a signature not present in the baseline are presented in a distinctive color reserved for new categories.
6. The method as recited in claim 1 , wherein causing presentation on a user interface (UI) of the detected anomaly further comprises:
presenting the count of log entries from the first plurality of log entries in the category and the count of log entries from the second plurality of log entries.
7. The method as recited in claim 1 , further comprising:
automatically performing an action based on information pertaining to the detected anomaly.
8. The method as recited in claim 1 , further comprising:
periodically recalculating the baseline based on a decay factor based on an age of the log entry.
9. The method as recited in claim 1 , further comprising:
calculating a working-hour baseline for log entries associated with working hours; and
calculating an off-hours baseline for long entries associated with non-working hours.
10. The method as recited in claim 1 , wherein causing presentation on a user interface (UI) of the detected anomaly further comprises presenting a difference between a count of logs in the baseline and a count of logs associated with the anomaly.
11. A system comprising:
a memory comprising instructions; and
one or more computer processors, wherein the instructions, when executed by the one or more computer processors, cause the system to perform operations comprising:
identifying a first plurality of log entries received over a predefined period of tune;
categorizing each log entry from the first plurality of log entries, into a category from a plurality of categories, based on a signature of the log entry, wherein the signature is based on a printf statement configured to generate log entries associated with the signature;
creating a baseline based on a number of log entries in each category, the baseline being a representative quantity received over the predefined period of time;
identifying a second plurality of log entries;
categorizing each log entry from the second plurality of log entries;
detecting an anomaly in one category from the plurality of categories based on a count of log entries from the second plurality of log entries in the category deviating from the count of log entries from the first plurality of log entries in the category by a predetermined threshold; and
causing presentation on a user interface (UI) of the detected anomaly, wherein causing presentation on a user interface (UI) of the detected anomaly further comprises:
presenting in the UI log entries with a signature not present in the baseline with a label indicating the log entries that are new; and
presenting logs missing from the baseline with a label of gone.
12. The system as recited in claim 11 , wherein the signature is associated with at least one static word in the log entry or at least one nonstatic portion corresponding to wild cards for words in the log entry, wherein the instructions further cause the one or more computer processors to perform operations comprising:
providing an option in the UI to edit the signature by adding wild cards in the signature.
13. The system as recited in claim 11 , wherein the instructions further cause the one or more computer processors to perform operations comprising:
causing presentation in the UI of a histogram based on the count of log entries, from the second plurality of log entries, for each unit of time granularity.
14. The system as recited in claim 11 , wherein the instructions further cause the one or more computer processors to perform operations comprising:
providing options in the UI for receiving information about the detected anomaly, the information comprising a description of the anomaly and a severity of the anomaly.
15. A tangible non-transitory machine-readable storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:
identifying a first plurality of log entries received over a predefined period of time;
categorizing each log entry from the first plurality of log entries, into a category from a plurality of categories, based on a signature of the log entry, wherein the signature is based on a printf statement configured to generate log entries associated with the signature;
creating a baseline based on a number of log entries in each category, the baseline being a representative quantity received over the predefined period of time;
identifying a second plurality of log entries;
categorizing each log entry from the second plurality of log entries;
detecting an anomaly in one category from the plurality of categories based on a count of log entries from the second plurality of log entries in the category deviating from the count of log entries from the first plurality of log entries in the category by a predetermined threshold; and
causing presentation on a user interface (UI) of the detected anomaly, wherein causing presentation on a user interface (LH) of the detected anomaly further comprises:
presenting in the UI log entries with a signature not present in the baseline with a label indicating the log entries that are new; and
presenting logs missing from the baseline with a label of gone.
16. The tangible non-transitory machine-readable storage medium as recited in claim 15 , wherein the signature is associated with at least one static word in the log entry or at least one non-static portion corresponding to wild cards for words in the log entry, wherein the machine further performs operations comprising:
providing an option in the UI to edit the signature by adding wild cards in the signature.
17. The tangible non-transitory machine-readable storage medium as recited in claim 15 , wherein the machine further performs operations comprising:
causing presentation in the UI of a histogram based on the count of log entries, from the second plurality of log entries, for each unit of time granularity.
18. The tangible non-transitory machine-readable storage medium as recited in claim 15 , wherein the machine further performs operations comprising:
providing options in the UI for receiving information about the detected anomaly, the information comprising a description of the anomaly and a severity of the anomaly.