Log analysis system employing long short-term memory recurrent neural net works
System logs are processed to identify and report anomalies in execution of processes of a log-generating system such as a data storage system. Log messages of system logs are vectorized to generate log-message vectors; long short-term memory (LSTM) neural network processing is applied to the log-message vectors to generate an LSTM output sequence representing a production flow of the processes; and second-level neural network processing is applied to a combination of the LSTM output sequence and a training sequence to generate an analysis sequence containing a representation of anomalies in the production flow of the processes, where the training sequence is generated from a non-anomalous training flow. An anomaly report is generated and provided to a report consumer for taking further action with respect to the anomalies represented in the analysis sequence.
1. A method of processing system logs to identify and report anomalies in execution of one or more processes of a log-generating computerized system, comprising:
vectorizing log messages of the system logs to generate corresponding log-message vectors;
applying long short-term memory (LSTM) neural network processing to the log-message vectors to generate an LSTM output sequence representing a production flow of the processes;
applying second-level neural network processing to a combination of the LSTM output sequence and a training sequence to generate an analysis sequence containing a representation of anomalies in the production flow of the processes, the training sequence generated from a non-anomalous training flow of the processes; and
based on the analysis sequence, generating an anomaly report and providing it to a report consumer for taking further action with respect to the anomalies represented in the analysis sequence.
2. The method of claim 1 , wherein the vectorizing is a multiple-level vectorization including (1) a word-level tokenization and vectorization of individual words in the log messages, which produces a first representation in which each log message is represented as a respective sequence of word vectors, and (2) a message-level vectorization of the sequence of word vectors of each log message to generate a respective log-message vector for each distinct log message, the log-message vectors forming a set of representations of the log messages in a multi-dimensional log-message vector space.
3. The method of claim 2 , wherein the log-message vectors produced by the multiple-level vectorization are segregated into respective clusters in the multi-dimensional log-message vector space, with each cluster containing log-message vectors for log messages belonging to a corresponding process flow, and further including cluster processing to extract distinct sequences of log-message vectors for the respective process flows and apply the sequences to respective LSTMs processing for detecting anomalies in the respective processes.
4. The method of claim 1 , wherein the log-message vectors produced by the vectorizing are segregated into respective clusters in the multi-dimensional log-message vector space, with each cluster containing log-message vectors for log messages belonging to a corresponding process flow, and further including cluster processing to extract distinct sequences of log-message vectors for the respective process flows and apply the sequences to respective LSTMs processing for detecting anomalies in the respective processes.
5. A log analyzer system having memory and processing circuitry, the memory storing computer program instructions executed by the processing circuitry to process system logs to identify and report anomalies in execution of one or more processes of a log-generating computerized system, the processing of the system logs including:
vectorizing log messages of the system logs to generate corresponding log-message vectors;
applying long short-term memory (LSTM) neural network processing to the log-message vectors to generate an LSTM output sequence representing a production flow of the processes;
applying second-level neural network processing to a combination of the LSTM output sequence and a training sequence to generate an analysis sequence containing a representation of anomalies in the production flow of the processes, the training sequence generated from a non-anomalous training flow of the processes; and
based on the analysis sequence, generating an anomaly report and providing it to a report consumer for taking further action with respect to the anomalies represented in the analysis sequence.
6. The log analyzer system of claim 5 , wherein the vectorizing is a multiple-level vectorization including (1) a word-level tokenization and vectorization of individual words in the log messages, which produces a first representation in which each log message is represented as a respective sequence of word vectors, and (2) a message-level vectorization of the sequence of word vectors of each log message to generate a respective log-message vector for each distinct log message, the log-message vectors forming a set of representations of the log messages in a multi-dimensional log-message vector space.
7. The log analyzer system of claim 6 , wherein the log-message vectors produced by the multiple-level vectorization are segregated into respective clusters in the multi-dimensional log-message vector space, with each cluster containing log-message vectors for log messages belonging to a corresponding process flow, and further including cluster processing to extract distinct sequences of log-message vectors for the respective process flows and apply the sequences to respective LSTMs processing for detecting anomalies in the respective processes.
8. The log analyzer system of claim 5 , wherein the log-message vectors produced by the vectorizing are segregated into respective clusters in the multi-dimensional log-message vector space, with each cluster containing log-message vectors for log messages belonging to a corresponding process flow, and further including cluster processing to extract distinct sequences of log-message vectors for the respective process flows and apply the sequences to respective LSTMs processing for detecting anomalies in the respective processes.