Content-level anomaly detection for heterogeneous logs
A computer-implemented method for automatically analyzing log contents received via a network and detecting content-level anomalies is presented. The computer-implemented method includes building a statistical model based on contents of a set of training logs and detecting, based on the set of training logs, content-level anomalies for a set of testing logs. The method further includes maintaining an index and metadata, generating attributes for fields, editing model capability to incorporate user domain knowledge, detecting anomalies using field attributes, and improving anomaly quality by using user feedback.
1. A computer-implemented method executed on a processor for automatically analyzing log contents received via a network and detecting content-level anomalies, the method comprising:
building a statistical model based on contents of a set of training logs; and
detecting, based on the set of training logs, content-level anomalies for a set of testing logs,
wherein the building of the statistical model includes parsing incoming logs into a plurality of fields.
2. The method of claim 1 , wherein each field of the plurality of fields includes an index where a key parameter refers to content and a value parameter refers to frequency count.
3. The method of claim 2 , wherein the following attributes are assigned to each field of the plurality of fields: an IGNORE attribute, a MEMBERSHIP_TEST attribute, and a RANGE_CHECK attribute.
4. The method of claim 3 , wherein if the attribute is the MEMBERSHIP_TEST attribute, it is determined if current field content is present in the statistical model's index.
5. The method of claim 3 , wherein if the attribute is the RANGE_CHECK attribute, it is determined if current field content is present within a range of the statistical model's index contents.
6. The method of claim 3 , wherein users are permitted to review assignments of the attributes and modify one or more of the assignments to incorporate their domain knowledge.
7. The method of claim 3 , wherein users are permitted to add and delete training log content to incorporate their domain knowledge.
8. A system for automatically analyzing log contents received via a network and detecting content-level anomalies, the system comprising:
a memory; and
a processor in communication with the memory, wherein the processor is configured to:
build a statistical model based on contents of a set of training logs; and
detect, based on the set of training logs, content-level anomalies for a set of testing logs,
wherein the building of the statistical model includes parsing incoming logs into a plurality of fields.
9. The system of claim 8 , wherein each field of the plurality of fields includes an index where a key parameter refers to content and a value parameter refers to frequency count.
10. The system of claim 9 , wherein the following attributes are assigned to each field of the plurality of fields: an IGNORE attribute, a MEMBERSHIP_TEST attribute, and a RANGE_CHECK attribute.
11. The system of claim 10 , wherein if the attribute is the MEMBERSHIP_TEST attribute, it is determined if current field content is present in the statistical model's index.
12. The system of claim 10 , wherein if the attribute is the RANGE_CHECK attribute, it is determined if current field content is present within a range of the statistical model's index contents.
13. The system of claim 10 , wherein users are permitted to review assignments of the attributes and modify one or more of the assignments to incorporate their domain knowledge.
14. The system of claim 10 , wherein users are permitted to add and delete training log content to incorporate their domain knowledge.
15. A non-transitory computer-readable storage medium comprising a computer-readable program for automatically analyzing log contents received via a network and detecting content-level anomalies, wherein the computer-readable program when executed on a computer causes the computer to perform the steps of:
building a statistical model based on contents of a set of training logs; and
detecting, based on the set of training logs, content-level anomalies for a set of testing logs,
wherein the building of the statistical model includes parsing incoming logs into a plurality of fields.
16. The non-transitory computer-readable storage medium of claim 15 , wherein each field of the plurality of fields including an index where a key parameter refers to content and a value parameter refers to frequency count.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the following attributes are assigned to each field of the plurality of fields: an IGNORE attribute, a MEMBERSHIP_TEST attribute, and a RANGE_CHECK attribute.
18. The non-transitory computer-readable storage medium of claim 17 ,
wherein if the attribute is the MEMBERSHIP_TEST attribute, it is determined if current field content is present in the statistical model's index; and
wherein if the attribute is the RANGE_CHECK attribute, it is determined if the current field content is present within a range of the statistical model's index contents.