Methods and systems for server failure prediction using server logs
Embodiments provide methods and systems of predicting server failures. A method may include accessing distinct log clusters representing instructions executed in server, applying first density machine learning model over input vector of distinct log clusters, with length equal to number of distinct log clusters, for obtaining first prediction output, applying first sequential machine learning model over time length sequence of distinct log clusters for obtaining second prediction output, applying second density machine learning model over input vector for obtaining third prediction output, applying second sequential machine learning model over time length sequence of distinct log clusters for obtaining fourth prediction output, aggregating first, second, third and fourth prediction outputs by ensemble model, and predicting likelihood of next log clusters to have anomalous behavior based on the aggregating. First density and first sequential models are trained by normal logs. Second density and second sequential models are trained by abnormal logs.
1 . A computer-implemented method, comprising:
accessing, by a processor, a set of distinct log clusters associated with a server, the set of distinct log clusters;
configuring, by the processor, a first density machine learning model based on a stacked autoencoder configured with a loss function that includes a Log-Cosh component and a constraint on a summation of frequency appearances of the set of distinct log clusters, the loss function structured to enable the autoencoder to reconstruct an input vector even when some log clusters are less frequent;
applying, by the processor, the first density machine learning model over the input vector associated with the set of distinct log clusters for obtaining a first prediction output, the input vector having a length equal to a number of the set of distinct log clusters, the first density machine learning model trained based on normal log data associated with the server to detect a first deviation of an input vector from an expected density distribution of log data that corresponds to normal operation, and generating the first prediction output based on the first deviation;
applying, by the processor, a first sequential machine learning model over a time length sequence of the set of distinct log clusters for obtaining a second prediction output, the first sequential machine learning model trained based on the normal log data associated with the server to detect a second deviation of the input vector from an expected sequence of the log data that corresponds to normal operation, and generating the second prediction output based on the second deviation;
wherein the first sequential machine learning model trained on the normal log data of the server is implemented using a Long Short Term Memory (LSTM) based sequential auto encoder;
applying, by the processor, a second density machine learning model over the input vector for obtaining a third prediction output, the second density machine learning model trained based on abnormal log data associated with the server to detect a third deviation of the input vector from an expected density distribution of the log data that corresponds to abnormal operation, and generating the third prediction output based on the third deviation;
applying, by the processor, a second sequential machine learning model over the time length sequence of the set of distinct log clusters for obtaining a fourth prediction output, the second sequential machine learning model trained based on the abnormal log data associated with the server to detect a fourth deviation of the input vector from an expected sequence of the log data that corresponds to abnormal operation, and generating the fourth prediction output based on the fourth deviation;
wherein the second density machine learning model trained on the abnormal log data of the server is implemented using a stacked auto encoder, and the second sequential machine learning model trained on the abnormal log data of the server is implemented using a Long Short Term Memory (LSTM) based sequential auto encoder;
aggregating, by the processor, the first, second, third and fourth prediction outputs based on an ensemble model; and
predicting, by the processor, a likelihood of next log clusters to have anomalous behavior based on the aggregating.
2 . The computer-implemented method of claim 1 , further comprising:
receiving, by the processor, a plurality of server logs from the server within a particular time interval; and
performing, by the processor, data pre-processing over the plurality of server logs for clustering the plurality of server logs into the set of distinct log clusters.
3 . The computer-implemented method of claim 2 , further comprising:
generating, by the processor, the input vector associated with the set of distinct log clusters, a value for each index of the input vector representing a frequency of appearances of each associated log cluster of the set of distinct log clusters during the particular time interval; and
identifying, by the processor, the time length sequence of the set of distinct log clusters processed during the particular time interval.
4 . The computer-implemented method of claim 1 , wherein aggregating the first, second, third and fourth prediction outputs based on the ensemble model comprises:
ensembling, by the processor, the first, second, third and fourth prediction outputs using a recurrent neural network model based at least on a weighted average ensemble method for obtaining an ensemble output.
5 . The computer-implemented method of claim 4 , further comprising:
comparing, by the processor, the ensemble output with a predetermined threshold value for determining the likelihood of the next log clusters to have the anomalous behavior; and
sending, by the processor, an alert to an administrator about future server failure occurrences based on the step of the predicting.
6 . An anomaly prediction system, comprising:
a communication interface;
a memory comprising executable instructions; and
a processor communicably coupled to the communication interface, the processor configured to execute the executable instructions to cause the anomaly prediction system to at least:
access a set of distinct log clusters associated with a server, the set of distinct log clusters representing instructions executed in the server;
configure a first density machine learning model based on a stacked autoencoder configured with a loss function that includes a Log-Cosh component and a constraint on a summation of frequency appearances of the set of distinct log clusters, the loss function structured to enable the autoencoder to reconstruct an input vector even when some log clusters are less frequent;
apply the first density machine learning model over the input vector associated with the set of distinct log clusters for obtaining a first prediction output, the input vector having a length equal to a number of the set of distinct log clusters, the first density machine learning model trained based on normal log data associated with the server, to detect a first deviation of an input vector from an expected density distribution of log data that corresponds to normal operation, and generating the first prediction output based on the first deviation;
apply a first sequential machine learning model over a time length sequence of the set of distinct log clusters for obtaining a second prediction output, the first sequential machine learning model trained based on the normal log data associated with the server, to detect a second deviation of the input vector from an expected sequence of the log data that corresponds to normal operation, and generating the second prediction output based on the second deviation;
wherein the first sequential machine learning model trained on the normal log data of the server is implemented using a Long Short Term Memory (LSTM) based sequential auto encoder;
apply a second density machine learning model over the input vector for obtaining a third prediction output, the second density machine learning model trained based on abnormal log data associated with the server, to detect a third deviation of the input vector from an expected density distribution of the log data that corresponds to abnormal operation, and generating the third prediction output based on the third deviation, apply a second sequential machine learning model over the time length sequence of the set of distinct log clusters for obtaining a fourth prediction output, the second sequential machine learning model trained based on the abnormal log data associated with the server, to detect a fourth deviation of the input vector from an expected sequence of the log data that corresponds to abnormal operation, and generating the fourth prediction output based on the fourth deviation;
wherein the second density machine learning model trained on the abnormal log data of the server is implemented using a stacked auto encoder, and the second sequential machine learning model trained on the abnormal log data of the server is implemented using a Long Short Term Memory (LSTM) based sequential auto encoder;
update, dynamically during a detection phase based on feedback indicating a misclassification, a probability weight distribution of: the first density machine learning model, the first sequential machine learning model, the second density machine learning model, and/or the second sequential machine learning model, thereby adapting to new log patterns to improve respective model performance;
aggregate the first, second, third and fourth prediction outputs based on an ensemble model; and
predict a likelihood of next log clusters to have anomalous behavior based on the aggregating.
7 . The anomaly prediction system of claim 6 , wherein the processor comprises:
a data pre-processing engine,
a first auto encoder comprising a first density auto encoder and a first sequential auto encoder,
a second auto encoder comprising a second density auto encoder and a second sequential auto encoder, an ensemble manager, and
a prediction engine.
8 . The anomaly prediction system of claim 7 , wherein the data pre-processing engine is configured to:
receive a plurality of server logs from the server within a particular time interval, and
perform data pre-processing over the plurality of server logs to cluster the plurality of server logs into the set of distinct log clusters.
9 . The anomaly prediction system of claim 8 , wherein the anomaly prediction system is further caused to:
generate the input vector associated with the set of distinct log clusters, a value for each index of the input vector representing a frequency of appearances of each associated log cluster of the set of distinct log clusters during the particular time interval, and
identify the time length sequence of the set of distinct log clusters processed during the particular time interval.
10 . The anomaly prediction system of claim 7 , wherein, to aggregate the first, second, third and fourth prediction outputs, the ensemble manager is configured to:
ensemble the first, second, third and fourth prediction outputs using a recurrent neural network model based at least on a weighted average ensemble method of obtaining an ensemble output.
11 . The anomaly prediction system of claim 10 , wherein the prediction engine is configured to:
compare the ensemble output with a predetermined threshold value to determine the likelihood of the next log clusters to have the anomalous behavior, and send an alert to an administrator about future server failure occurrence based on the comparison.
12 . A computer-implemented method, comprising:
accessing, by a processor, a set of distinct log clusters associated with a server, the set of distinct log clusters representing instructions executed in the server;
configuring, by the processor, a first density machine learning model based on a stacked autoencoder configured with a loss function that includes a Log-Cosh component and a constraint on a summation of frequency appearances of the set of distinct log clusters, the loss function structured to enable the autoencoder to reconstruct an input vector even when some log clusters are less frequent;
applying, by the processor, the first density machine learning model over the input vector associated with the set of distinct log clusters for obtaining a first prediction output, the input vector having a length equal to a number of the set of distinct log clusters, the first density machine learning model trained based on normal log data associated with the server to detect a first deviation of an input vector from an expected density distribution of log data that corresponds to normal operation, and generating the first prediction output based on the first deviation;
applying, by the processor, a first sequential machine learning model over a time length sequence of the set of distinct log clusters for obtaining a second prediction output, the first sequential machine learning model trained based on the normal log data associated with the server to detect a second deviation of the input vector from an expected sequence of the log data that corresponds to normal operation, and generating the second prediction output based on the second deviation;
wherein the first sequential machine learning model trained on the normal log data of the server is implemented using a Long Short Term Memory (LSTM) based sequential auto encoder;
generating, by the processor, an ensemble output of the first prediction output and the second prediction output for identifying whether server anomalies are detected or not; and in response to identification of the server anomalies, performing:
applying, by the processor, a second density machine learning model over the input vector for obtaining a third prediction output, the second density machine learning model trained based on abnormal log data associated with the server to detect a third deviation of the input vector from an expected density distribution of the log data that corresponds to abnormal operation, and generating the third prediction output based on the third deviation;
applying, by the processor, a second sequential machine learning model over the time length sequence of the set of distinct log clusters for obtaining a fourth prediction output, the second sequential machine learning model trained based on the abnormal log data associated with the server to detect a fourth deviation of the input vector from an expected sequence of the log data that corresponds to abnormal operation, and generating the fourth prediction output based on the fourth deviation;
wherein the second density machine learning model trained on the abnormal log data of the server is implemented using a stacked auto encoder, and the second sequential machine learning model trained on the abnormal log data of the server is implemented using a Long Short Term Memory (LSTM) based sequential auto encoder;
updating, by the processor, dynamically during a detection phase based on feedback indicating a misclassification, a probability weight distribution of: the first density machine learning model, the first sequential machine learning model, the second density machine learning model, and/or the second sequential machine learning model, thereby adapting to new log patterns to improve respective model performance;
generating, by the processor, a second ensemble output of the third prediction output and the fourth prediction output based at least part on an ensemble model;
predicting, by the processor, a likelihood of next log clusters to have anomalous behavior based on the second ensemble output.
13 . The computer-implemented method of claim 12 , further comprising:
generating, by the processor, the input vector associated with the set of distinct log clusters, a value for each index of the input vector representing a frequency of appearances of each associated log cluster of the set of distinct log clusters during a particular time interval; and
identifying, by the processor, the time length sequence of the set of distinct log clusters processed during the particular time interval.
14 . The computer-implemented method of claim 12 , wherein predicting the likelihood of the next log clusters to have the anomalous behavior based on the second ensemble output comprises:
comparing the second ensemble output with a predetermined threshold value for determining the likelihood of the next log clusters to have the anomalous behavior; and
sending an alert to an administrator about future server failure occurrences based at least on a step of the comparing.