Log-based system maintenance and management
Methods and systems for system maintenance include identifying patterns in heterogeneous logs. Predictive features are extracted from a set of input logs based on the identified patterns. It is determined that the predictive features indicate a future system failure using a first model. A second model is trained, based on a target sample from the predictive features and based on weights associated with a distance between the target sample and a set of samples from the predictive features, to identify one or more parameters of the second model associated with the future system failure. A system maintenance action is performed in accordance with the identified one or more parameters.
1. A method for system maintenance, comprising:
identifying patterns in heterogeneous logs;
extracting predictive features from a set of input logs based on the identified patterns;
determining that the predictive features indicate a future system failure using a first model;
training a second model, based on a target sample from the predictive features and based on weights associated with a distance between the target sample and a plurality of samples from the predictive features, to identify one or more parameters of the second model associated with the future system failure, wherein training the second model includes setting the weights according to an objective function:
arg
min
(
1
n
∑
i
=
1
n
(
〈
X
i
,
β
〉
-
Y
i
)
2
+
γ
B
l
1
d
)
where X i is a term frequency inverse document frequency (TF-IDF) feature for an i th input log, Y i is a corresponding prediction, γ is a regularization parameter, n is a number of input logs, β is a vector of the weights, and ∥β∥ l 1 d is an L1 norm of β in a d-dimensional Euclidean space; and
performing a system maintenance action in accordance with the identified one or more parameters.
2. The method of claim 1 , wherein the first model is a combination of a long-short-term memory recurrent neural network and a gradient-boosting decision tree.
3. The method of claim 2 , wherein determining that the predictive features indicate a future system failure comprises generating a first failure likelihood from the long-short-term memory recurrent neural network and a second failure likelihood from the gradient-boosting decision tree.
4. The method of claim 3 , wherein determining that the predictive features indicate a future system failure further comprises averaging the first failure likelihood and the second failure likelihood to generate a combined failure likelihood and comparing the combined failure likelihood to a failure prediction threshold.
5. The method of claim 1 , wherein the second model is a sparse L1-regularized linear model.
6. The method of claim 1 , wherein identifying patterns in heterogeneous logs comprises clustering the heterogeneous logs according to similarity.
7. The method of claim 1 , wherein extracting predictive features comprises converting pattern distribution features according to Term-Frequency Inverse-Document-Frequency (TF-IDF).
8. The method of claim 7 , wherein extracting predictive features comprises further reducing dimensionality of the converted pattern distribution features.
9. The method of claim 1 , wherein performing the system maintenance action comprises performing an action selected from the group consisting of adjusting an environmental control, shutting down a device at risk for a hardware failure, and changing a system security policy.
10. A method for system maintenance, comprising:
identifying patterns in heterogeneous logs by clustering the heterogeneous logs according to similarity;
extracting predictive features from a set of input logs based on the identified patterns by converting pattern distribution features according to Term-Frequency Inverse-Document-Frequency (TF-IDF);
determining that the predictive features indicate a future system failure using a first model that averages an output of two distinct machine learning models;
training a second model, based on a target sample from the predictive features and based on weights associated with a distance between the target sample and a plurality of samples from the predictive features, to identify one or more parameters of the second model associated with the future system failure, wherein training the second model includes setting the weights according to an objective function:
arg
min
(
1
n
∑
i
=
1
n
(
〈
X
i
,
β
〉
-
Y
i
)
2
+
γ
B
l
1
d
)
where X i is a term frequency inverse document frequency (TF-IDF) feature for an i th input log, Y i is a corresponding prediction, γ is a regularization parameter, n is a number of input logs, β is a vector of the weights, and ∥β∥ l 1 d is an L1 norm of β in a d-dimensional Euclidean space; and
performing a system maintenance action in accordance with the identified one or more parameters.
11. The method of claim 10 , wherein performing the system maintenance action comprises performing an action selected from the group consisting of adjusting an environmental control, shutting down a device at risk for a hardware failure, and changing a system security policy.
12. A system for system maintenance, comprising:
a log pattern module configured to identify patterns in heterogeneous logs;
a feature extraction module configured to extract predictive features from a set of input logs based on the identified patterns;
a failure prediction module configured to determine that the predictive features indicate a future system failure using a first model;
an interpretation module configured to train a second model, based on a target sample from the predictive features and based on weights associated with a distance between the target sample and a plurality of samples from the predictive features, to identify one or more parameters of the second model associated with the future system failure, wherein the interpretation module trains the second model by setting the weights according to an objective function:
arg
min
(
1
n
∑
i
=
1
n
(
〈
X
i
,
β
〉
-
Y
i
)
2
+
γ
B
l
1
d
)
where X i is a term frequency inverse document frequency (TF-IDF) feature for an i th input log, Y i is a corresponding prediction, γ is a regularization parameter, n is a number of input logs, β is a vector of the weights, and ∥β∥ l 1 d is an L1 norm of β in a d-dimensional Euclidean space; and
a maintenance module configured to perform a system maintenance action in accordance with the identified one or more parameters.
13. The system of claim 12 , wherein the first model is a combination of a long-short-term memory recurrent neural network and a gradient-boosting decision tree.
14. The system of claim 13 , wherein the failure prediction module is further configured to generate a first failure likelihood from the long-short-term memory recurrent neural network and a second failure likelihood from the gradient-boosting decision tree.
15. The system of claim 14 , wherein the failure prediction module is further configured to average the first failure likelihood and the second failure likelihood to generate a combined failure likelihood and to compare the combined failure likelihood to a failure prediction threshold.
16. The system of claim 12 , wherein the second model is a sparse L1-regularized linear model.
17. The system of claim 12 , the log pattern module is further configured to cluster the heterogeneous logs according to similarity.
18. The system of claim 12 , wherein the feature extraction module is further configured to convert pattern distribution features according to Term-Frequency Inverse-Document-Frequency (TF-IDF).
19. The system of claim 18 , wherein the feature extraction module is further configured to reduce dimensionality of the converted pattern distribution features.
20. The system of claim 12 , wherein the maintenance module is further configured to perform an action selected from the group consisting of adjusting an environmental control, shutting down a device at risk for a hardware failure, and changing a system security policy.