System and method for managing operation of data processing systems to meet operational goals
Methods and systems for managing data processing systems are disclosed. A data processing system may include and depend on the operation of hardware and/or software components. To manage the operation of the data processing system, a data processing system manager may obtain logs for components of the data processing system. Inference models may be implemented to predict likely future component failures (e.g., failure sequences) and their associated times-to-failures using information recorded in the logs. The failure sequences may be presented as an acyclic graph that associates component failures, their times-to-failure, and related actions. The probable failure sequences may be analyzed to identify sets of actions. The sets of actions may be optimized based on deviations detected during executing the sets of actions to optimize operational goals (e.g., maximizing system lifetime, minimizing system costs), and/or reduce the likelihood of the data processing system becoming impaired.
1 . A method for managing data processing systems, comprising:
obtaining, by a data processing system manager of the data processing systems, actual data in real-time while a manageable data processing system is simultaneously being caused by the data processing system manager to execute a failure-related action associated with an acyclic graph obtained by the data processing system manager to remediate a current failure or to prevent a potential future failure of the manageable data processing system, wherein the manageable data processing system is one of the data processing systems and the actual data is associated with operations performed and data generated by the manageable data processing system as the manageable data processing system executes the failure-related action, and the acyclic graph comprises a plurality of nodes connected by a plurality of edges where the plurality of edges represent relationships between the plurality of nodes and the relationships being probabilistically defined based on historical failure information for the data processing systems;
determining, by the data processing system manager, that a deviation exists between the actual data and expected data specified in the acyclic graph;
calculating, by the data processing system manager and based on determining that the deviation exists, a match score between the actual data and the expected data;
determining, by the data processing system manager and based on the match score, a course of action associated with a configuring of the manageable data processing system, wherein the course of action is associated with achieving an operational goal of the manageable data processing system and the deviation decreases a likelihood of the manageable data processing system meeting the operational goal; and
causing, by the data processing system manager and through sending instructions to the manageable data processing system via a communication system that operably connects the data processing system manager to the manageable data processing system, the manageable data processing system to execute the course of action,
wherein executing the course of action prevents the decreases in the likelihood of the manageable data processing system meeting the operation goal, and
wherein the method further comprises, and by the data processing system manager before obtaining the actual data:
obtaining the acyclic graph based on the historical failure information for the data processing systems;
identifying the operational goal for the manageable data processing system, the operational goal being associated with remediation of the current failure or prevention of the potential future failure of the manageable data processing system;
analyzing the acyclic graph based on the operational goal to obtain an action set by:
enumerating traversal paths through the acyclic graph;
for each traversal path of the enumerated traversal paths:
obtaining an aggregate relationship based on the relationships associated with all of the edges along the traversal path,
obtaining a fitness value based on the aggregate relationship and the operational goal, and
obtaining a rank for the traversal path based on the fitness value, the rank being usable to order the traversal paths;
selecting an initial best fit traversal path of the traversal paths based on the rank of each of the traversal paths;
obtaining the action set based on the initial best fit traversal path; and
causing the manageable data processing system to execute the action set as the failure-related action.
2 . The method of claim 1 , wherein obtaining the acyclic graph comprises and by the data processing system manager:
establishing a second node based on a failure of a second component type, the failure of the second component type occurring after a failure of a first component type, and the failure of the first component type being represented by a first node;
establishing a first edge between the first node and the second node, the first edge being associated with a first relationship;
establishing a third node based on a failure of a third component type, the failure of the third component type occurring after the failure of the second component type; and
establishing a second edge between the second node and the third node, the second edge being associated with a second relationship.
3 . The method of claim 1 , wherein determining the course of action comprises:
generating, based on the deviation and the traversal paths of the acyclic graph, a new action set including one or more new actions different from one or more original actions that make up the action set, the new action set being the course of action.
4 . The method of claim 3 , wherein the new action set is associated with a new best-fit traversal path among the traversal paths that is different from the initial best fit traversal path.
5 . The method of claim 1 , wherein the expected data is associated with one or more expected operations the manageable data processing system should perform and expected data the manageable data processing system should generate as the manageable data processing system executes the failure-related action, and the deviation causes the manageable data processing system to fail to remediate the current failure or prevent the potential future failure.
6 . The method of claim 1 , wherein the match score is based on:
a similarity value between the actual data and the expected data, and
an attribution score assigned to the expected data.
7 . The method of claim 1 , wherein the acyclic graph is obtained from a storage of the data processing system manager.
8 . A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor of a data processing system manager that manages data processing systems, cause the data processing system manager to perform operations for managing the data processing systems, the operations comprising:
obtaining actual data in real-time while a manageable data processing system is simultaneously being caused by the data processing system manager to execute a failure-related action associated with an acyclic graph obtained by the data processing system manager to remediate a current failure or to prevent a potential future failure of the manageable data processing system, wherein the manageable data processing system is one of the data processing systems and the actual data is associated with operations performed and data generated by the manageable data processing system as the manageable data processing system executes the failure-related action, and the acyclic graph comprises a plurality of nodes connected by a plurality of edges where the plurality of edges represent relationships between the plurality of nodes and the relationships being probabilistically defined based on historical failure information for the data processing systems;
determining that a deviation exists between the actual data and expected data specified in the acyclic graph;
calculating, based on determining that the deviation exists, a match score between the actual data and the expected data;
determining, by the data processing system manager and based on the match score, a course of action associated with a configuring of the manageable data processing system, wherein the course of action is associated with achieving an operational goal of the manageable data processing system and the deviation decreases a likelihood of the manageable data processing system meeting the operational goal; and
causing, by sending instructions to the manageable data processing system via a communication system that operably connects the data processing system manager to the manageable data processing system, the manageable data processing system to execute the course of action,
wherein executing the course of action prevents the decreases in the likelihood of the manageable data processing system meeting the operation goal, and
wherein the operations further comprise, before obtaining the actual data:
obtaining the acyclic graph based on the historical failure information for the data processing systems;
identifying the operational goal for the manageable data processing system, the operational goal being associated with remediation of the current failure or prevention of the potential future failure of the manageable data processing system;
analyzing the acyclic graph based on the operational goal to obtain an action set by:
enumerating traversal paths through the acyclic graph;
for each traversal path of the enumerated traversal paths:
obtaining an aggregate relationship based on the relationships associated with all of the edges along the traversal path,
obtaining a fitness value based on the aggregate relationship and the operational goal, and
obtaining a rank for the traversal path based on the fitness value, the rank being usable to order the traversal paths;
selecting an initial best fit traversal path of the traversal paths based on the rank of each of the traversal paths; and
obtaining the action set based on the initial best fit traversal path; and
causing the manageable data processing system to execute the action set as the failure-related action.
9 . The non-transitory machine-readable medium of claim 8 , wherein obtaining the acyclic graph comprises:
establishing a second node based on a failure of a second component type, the failure of the second component type occurring after a failure of a first component type, and the failure of the first component type being represented by a first node;
establishing a first edge between the first node and the second node, the first edge being associated with a first relationship;
establishing a third node based on a failure of a third component type, the failure of the third component type occurring after the failure of the second component type; and
establishing a second edge between the second node and the third node, the second edge being associated with a second relationship.
10 . The non-transitory machine-readable medium of claim 8 , wherein determining the course of action comprises:
generating, based on the deviation and the traversal paths of the acyclic graph, a new action set including one or more new actions different from one or more original actions that make up the action set, the new action set being the course of action.
11 . The non-transitory machine-readable medium of claim 10 , wherein the new action set is associated with a new best-fit traversal path among the traversal paths that is different from the initial best fit traversal path.
12 . The non-transitory machine-readable medium of claim 8 , wherein the expected data is associated with one or more expected operations the manageable data processing system should perform and expected data the manageable data processing system should generate as the manageable data processing system executes the failure-related action, and the deviation causes the manageable data processing system to fail to remediate the current failure or prevent the potential future failure.
13 . The non-transitory machine-readable medium of claim 8 , wherein the match score is based on:
a similarity value between the actual data and the expected data, and
an attribution score assigned to the expected data.
14 . The non-transitory machine-readable medium of claim 8 , wherein the acyclic graph is obtained from a storage of the data processing system manager.
15 . A data processing system manager, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations for managing data processing systems, the operations comprising:
obtaining actual data in real-time while a manageable data processing system is simultaneously being caused by the data processing system manager to execute a failure-related action associated with an acyclic graph obtained by the data processing system manager to remediate a current failure or to prevent a potential future failure of the manageable data processing system, wherein the manageable data processing system is one of the data processing systems and the actual data is associated with operations performed and data generated by the manageable data processing system as the manageable data processing system executes the failure-related action, and the acyclic graph comprises a plurality of nodes connected by a plurality of edges where the plurality of edges represent relationships between the plurality of nodes and the relationships being probabilistically defined based on historical failure information for the data processing systems;
determining that a deviation exists between the actual data and expected data specified in the acyclic graph;
calculating, based on determining that the deviation exists, a match score between the actual data and the expected data;
determining, based on the match score, a course of action associated with a configuring of the manageable data processing system, wherein the course of action is associated with achieving an operational goal of the manageable data processing system and the deviation decreases a likelihood of the manageable data processing system meeting the operational goal; and
causing, by sending instructions to the manageable data processing system via a communication system that operably connects the data processing system manager to the manageable data processing system, the manageable data processing system to execute the course of action,
wherein executing the course of action prevents the decreases in the likelihood of the manageable data processing system meeting the operation goal, and
wherein the operations further comprise, before obtaining the actual data:
obtaining the acyclic graph based on the historical failure information for the data processing systems;
identifying the operational goal for the manageable data processing system, the operational goal being associated with remediation of the current failure or prevention of the potential future failure of the manageable data processing system;
analyzing the acyclic graph based on the operational goal to obtain an action set by:
enumerating traversal paths through the acyclic graph;
for each traversal path of the enumerated traversal paths:
obtaining an aggregate relationship based on the relationships associated with all of the edges along the traversal path,
obtaining a fitness value based on the aggregate relationship and the operational goal, and
obtaining a rank for the traversal path based on the fitness value, the rank being usable to order the traversal paths;
selecting an initial best fit traversal path of the traversal paths based on the rank of each of the traversal paths; and
obtaining the action set based on the initial best fit traversal path; and
causing the manageable data processing system to execute the action set as the failure-related action.
16 . The data processing system manager of claim 15 , wherein obtaining the acyclic graph comprises and by the data processing system manager:
establishing a second node based on a failure of a second component type, the failure of the second component type occurring after a failure of a first component type, and the failure of the first component type being represented by a first node;
establishing a first edge between the first node and the second node, the first edge being associated with a first relationship;
establishing a third node based on a failure of a third component type, the failure of the third component type occurring after the failure of the second component type; and
establishing a second edge between the second node and the third node, the second edge being associated with a second relationship.
17 . The data processing system manager of claim 15 , wherein determining the course of action comprises:
generating, based on the deviation and the traversal paths of the acyclic graph, a new action set including one or more new actions different from one or more original actions that make up the action set, the new action set being the course of action.
18 . The data processing system manager of claim 17 , wherein the new action set is associated with a new best-fit traversal path among the traversal paths that is different from the initial best fit traversal path.
19 . The data processing system manager of claim 15 , wherein the expected data is associated with one or more expected operations the manageable data processing system should perform and expected data the manageable data processing system should generate as the manageable data processing system executes the failure-related action, and the deviation causes the manageable data processing system to fail to remediate the current failure or prevent the potential future failure.
20 . The data processing system manager of claim 15 , wherein the match score is based on:
a similarity value between the actual data and the expected data, and
an attribution score assigned to the expected data.