Identifying and remediating system anomalies through machine learning algorithms
Methods, apparatus, and processor-readable storage media for identifying and remediating anomalies through cognitively assorted machine learning algorithms are provided herein. A computer-implemented method includes: identifying, using system log data, a target variable based at least in part on correlations between a set of performance indicators of a system and the target variable, and threshold values for the performance indicators relative to the target variable; generating an inference model to predict when the system will enter an adverse state and identify one or more root causes of the system entering the adverse state; using machine reinforcement learning to determine an action policy including actions that remediate the adverse state; predicting that the system will enter the adverse state by applying the inference model to further system log data; and automatically executing one or more actions of the action policy in response to the prediction.
1. A computer-implemented method comprising:
identifying, using a first set of system log data, a target variable based at least in part on (i) one or more linear correlations between a set of performance indicators of a system and the target variable, and (ii) one or more threshold values for each of the performance indicators relative to the target variable;
generating an inference model to (i) predict when the system will enter one or more adverse states based at least on a given value of the target variable and (ii) identify one or more root causes of the system entering the one or more adverse states based on at least a subset of the set of performance indicators, wherein said generating comprises:
generating a linear model by implementing logistic regression analysis based at least in part on the target variable and the at least a subset of the performance indicators using the first set of system log data;
generating a non-linear model by implementing a random forest of decision trees for the at least a subset of the performance indicators using the first set of system log data; and
combining the linear model and the non-linear model, to form the inference model, using at least one ensemble modeling technique;
using machine reinforcement learning to determine an action policy comprising one or more actions that at least partially remediate the one or more adverse states;
predicting that the system will enter at least one of the adverse states by applying the inference model to a second set of system log data; and
automatically executing one or more actions of the action policy in response to said predicting;
wherein the method is performed by at least one processing device comprising a processor coupled to a memory.
2. The computer-implemented method of claim 1 , further comprising:
determining the one or more threshold values by applying a k-means algorithm for each performance indicator to the first set of system log data.
3. The computer-implemented method of claim 2 , further comprising:
labeling the first set of system log data based at least in part on the determined one or more threshold values to create a training dataset, wherein generating the linear model and the non-linear model comprises training the linear model and the non-linear model using at least a portion of the training dataset.
4. The computer-implemented method of claim 1 , wherein the first set of system log data and the second set of system log data comprise one or more events, and wherein said predicting comprises determining a state of the system for each of the one or more events.
5. The computer-implemented method of claim 4 , wherein the first set of system log data and the second set of system log data comprise at least one of:
data from a log corresponding to at least one application implemented on a service-oriented architecture (SOA) platform of the system; and
data from a log from at least one virtual machine corresponding to the system.
6. The computer-implemented method of claim 1 , further comprising:
using the machine reinforcement learning to update the action policy in response to performance of at least one of the actions.
7. The computer-implemented method of claim 1 , wherein the system comprises a distributed file system.
8. The computer-implemented method of claim 1 , wherein the one or more adverse states affect one or more of availability and performance of the system.
9. The computer-implemented method of claim 1 , wherein the set of performance indicators comprises one or more of:
at least one performance indicator corresponding to usage of a processor;
at least one performance indicator corresponding to availability of a processor;
at least one performance indicator corresponding to a response time;
at least one performance indicator corresponding to a number of timeouts associated with a service-oriented architecture platform; and
at least one performance indicator corresponding to a number of errors associated with a service-oriented architecture platform.
10. The computer-implemented method of claim 1 , further comprising:
outputting an identification of one or more of the root causes corresponding to the at least one adverse state associated with said predicting.
11. A non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device:
to identify, using a first set of system log data, a target variable based at least in part on (i) one or more linear correlations between a set of performance indicators of a system and the target variable, and (ii) one or more threshold values for each of the performance indicators relative to the target variable;
to generate an inference model to (i) predict when the system will enter one or more adverse states based at least on a given value of the target variable and (ii) identify one or more root causes of the system entering the one or more adverse states based on at least a subset of the set of performance indicators, wherein said generating comprises:
generating a linear model by implementing logistic regression analysis based at least in part on the target variable and the at least a subset of the performance indicators using the first set of system log data;
generating a non-linear model by implementing a random forest of decision trees for the at least a subset of the performance indicators using the first set of system log data; and
combining the linear model and the non-linear model, to form the inference model, using at least one ensemble modeling technique;
to use machine reinforcement learning to determine an action policy comprising one or more actions that at least partially remediate the one or more adverse states;
to predict that the system will enter at least one of the adverse states by applying the inference model to a second set of system log data; and
to automatically execute one or more actions of the action policy in response to said prediction.
12. The non-transitory processor-readable storage medium of claim 11 , wherein the program code when executed by at least one processing device causes the at least one processing device:
to determine the one or more threshold values by applying a k-means algorithm for each performance indicator to the first set of system log data.
13. The non-transitory processor-readable storage medium of claim 11 , wherein the program code when executed by at least one processing device causes the at least one processing device:
to use the machine reinforcement learning to update the action policy in response to performance of at least one of the actions.
14. An apparatus comprising:
at least one processing device comprising a processor coupled to a memory;
the at least one processing device being configured:
to identify, using a first set of system log data, a target variable based at least in part on (i) one or more linear correlations between a set of performance indicators of a system and the target variable, and (ii) one or more threshold values for each of the performance indicators relative to the target variable;
to generate an inference model to (i) predict when the system will enter one or more adverse states based at least on a given value of the target variable and (ii) identify one or more root causes of the system entering the one or more adverse states based on at least a subset of the set of performance indicators, wherein said generating comprises:
generating a linear model by implementing logistic regression analysis based at least in part on the target variable and the at least a subset of the performance indicators using the first set of system log data;
generating a non-linear model by implementing a random forest of decision trees for the at least a subset of the performance indicators using the first set of system log data; and
combining the linear model and the non-linear model, to form the inference model, using at least one ensemble modeling technique;
to use machine reinforcement learning to determine an action policy comprising one or more actions that at least partially remediate the one or more adverse states;
to predict that the system will enter at least one of the adverse states by applying the inference model to a second set of system log data; and
to automatically execute one or more actions of the action policy in response to said prediction.
15. The apparatus of claim 14 , wherein the at least one processing device being further configured:
to determine the one or more threshold values by applying a k-means algorithm for each performance indicator to the first set of system log data.
16. The apparatus of claim 14 , wherein the first set of system log data and the second set of system log data comprise one or more events, and wherein said predicting comprises determining a state of the system for each of the one or more events.
17. The apparatus of claim 16 , wherein the first set of system log data and the second set of system log data comprise at least one of:
data from a log corresponding to at least one application implemented on a service-oriented architecture (SOA) platform of the system; and
data from a log from at least one virtual machine corresponding to the system.
18. The apparatus of claim 14 , wherein the system comprises a distributed file system.
19. The apparatus of claim 14 , wherein the one or more adverse states affect one or more of availability and performance of the system.
20. The apparatus of claim 14 , wherein the set of performance indicators comprises one or more of:
at least one performance indicator corresponding to usage of a processor;
at least one performance indicator corresponding to availability of a processor;
at least one performance indicator corresponding to a response time;
at least one performance indicator corresponding to a number of timeouts associated with a service-oriented architecture platform; and
at least one performance indicator corresponding to a number of errors associated with a service-oriented architecture platform.