Graph-based log sequence anomaly detection and problem diagnosis
Techniques include generating a log sequence for new logs that have been received, searching a log sequence database for the log sequence having been generated, and determining that the log sequence is anomalous in response to not finding an identical log sequence in the log sequence database. In response to the log sequence not being found in the log sequence database, the log sequence is compared to a graph of historical log sequences to find a closest sequence path to one or more historical log sequences. An anomaly of the log sequence is diagnosed based on an occurrence at which the log sequence deviates from the closest sequence path of the one or more historical log sequences.
1. A computer-implemented method comprising:
generating a log sequence for new logs that have been received;
generating a vector representation of the log sequence, wherein a neural network is used to generate the vector representation, wherein the neural network is trained on historical log sequences to generate vector representations of the historical log sequences;
searching a log sequence database for the vector representation of the log sequence;
determining that the log sequence is anomalous in response to not finding an identical vector representation for a corresponding log sequence in the log sequence database;
in response to the vector representation of the log sequence not being found in the log sequence database, comparing the log sequence to a graph of historical log sequences to find a closest sequence path to one or more historical log sequences; and
diagnosing an anomaly of the log sequence based on an occurrence at which the log sequence deviates from the closest sequence path of the one or more historical log sequences.
2. The computer-implemented method of claim 1 , wherein the log sequence comprises a sequence of labels corresponding to message identifications of log messages in the new logs.
3. The computer-implemented method of claim 1 , wherein the occurrence at which the log sequence deviates from the closest sequence path identifies a label in the log sequence.
4. The computer-implemented method of claim 3 , wherein the label having been identified corresponds to a log message in the new logs, thereby diagnosing the log message as the anomaly.
5. The computer-implemented method of claim 1 , wherein the occurrence at which the log sequence deviates from the closest sequence path identifies one or more nodes in the closest sequence path of the graph not found in an order of labels of the log sequence.
6. The computer-implemented method of claim 5 , wherein the one or more nodes in the closest sequence path of the graph not found in the order of the labels of the log sequence are candidates for correcting the anomaly.
7. The computer-implemented method of claim 1 , wherein the graph of the historical log sequences comprises nodes connected to one another by edges, each of the nodes corresponding to a label of the historical log sequences, the edges following a sequential order of the historical log sequences.
8. A system comprising:
a memory having computer readable instructions; and
one or more processors for executing the computer readable instructions, the computer readable instructions controlling the one or more processors to perform operations comprising:
generating a log sequence for new logs that have been received;
generating a vector representation of the log sequence, wherein a neural network is used to generate the vector representation, wherein the neural network is trained on historical log sequences to generate vector representations of the historical log sequences;
searching a log sequence database for the vector representation of the log sequence;
determining that the log sequence is anomalous in response to not finding an identical vector representation for a corresponding log sequence in the log sequence database;
in response to the vector representation of the log sequence not being found in the log sequence database, comparing the log sequence to a graph of historical log sequences to find a closest sequence path to one or more historical log sequences; and
diagnosing an anomaly of the log sequence based on an occurrence at which the log sequence deviates from the closest sequence path of the one or more historical log sequences.
9. The system of claim 8 , wherein the log sequence comprises a sequence of labels corresponding to message identifications of log messages in the new logs.
10. The system of claim 8 , wherein the occurrence at which the log sequence deviates from the closest sequence path identifies a label in the log sequence.
11. The system of claim 10 , wherein the label having been identified corresponds to a log message in the new logs, thereby diagnosing the log message as the anomaly.
12. The system of claim 8 , wherein the occurrence at which the log sequence deviates from the closest sequence path identifies one or more nodes in the closest sequence path of the graph not found in an order of labels of the log sequence.
13. The system of claim 12 , wherein the one or more nodes in the closest sequence path of the graph not found in the order of the labels of the log sequence are candidates for correcting the anomaly.
14. The system of claim 8 , wherein the graph of the historical log sequences comprises nodes connected to one another by edges, each of the nodes corresponding to a label of the historical log sequences, the edges following a sequential order of the historical log sequences.
15. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform operations comprising:
generating a log sequence for new logs that have been received;
generating a vector representation of the log sequence, wherein a neural network is used to generate the vector representation, wherein the neural network is trained on historical log sequences to generate vector representations of the historical log sequences;
searching a log sequence database for the vector representation of the log sequence;
determining that the log sequence is anomalous in response to not finding an identical the vector representation for a corresponding log sequence in the log sequence database;
in response to the vector representation of the log sequence not being found in the log sequence database, comparing the log sequence to a graph of historical log sequences to find a closest sequence path to one or more historical log sequences; and
diagnosing an anomaly of the log sequence based on an occurrence at which the log sequence deviates from the closest sequence path of the one or more historical log sequences.
16. The computer program product of claim 15 , wherein the log sequence comprises a sequence of labels corresponding to message identifications of log messages in the new logs.
17. The computer program product of claim 15 , wherein the occurrence at which the log sequence deviates from the closest sequence path identifies a label in the log sequence.
18. The computer program product of claim 17 , wherein the label having been identified corresponds to a log message in the new logs, thereby diagnosing the log message as the anomaly.
19. The computer program product of claim 15 , wherein the occurrence at which the log sequence deviates from the closest sequence path identifies one or more nodes in the closest sequence path of the graph not found in an order of labels of the log sequence.
20. The computer program product of claim 19 , wherein the one or more nodes in the closest sequence path of the graph not found in the order of the labels of the log sequence are candidates for correcting the anomaly.