Method for finding failing components in a large distributed storage system connectivity
View Patent ↗Embodiments of the present systems and methods may provide techniques for finding failing components in a distributed storage system. For example a method may comprise measuring problems and health of a plurality of physical and logical components in a distributed storage system, the plurality of physical and logical components forming nodes of the distributed storage system, and generating a graph of the nodes organized in a plurality of hierarchical levels, generating, for each node in the graph, a score summarizing the measured problems and health of the node, determining a highest score at a highest hierarchical level of the graph and determining the associated node as a failing component at a most significant level.
1. A method implemented in a computer comprising a processor, memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor, the method comprising:
measuring problems and health of a plurality of physical and logical components in a distributed storage system, the plurality of physical and logical components forming nodes of the distributed storage system, and generating a graph of the nodes organized in a plurality of hierarchical levels;
generating, for each node in the graph, a score summarizing the measured problems and health of the node;
determining a highest score at a highest hierarchical level of the graph by:
traversing the nodes of the plurality of hierarchical levels of the graph, starting at a highest hierarchical level,
determining a node having a score that is greater than or equal to scores of any nodes in a sub-graph of the node at any lower level, and
determining that the determined node is a failing component at a most significant level; and
determining an associated node as a failing component at a most significant level.
2. The method of claim 1 , further comprising generating a list of additional failing components, the list ordered by a highest score of each successive node at each successive hierarchical level of the graph.
3. The method of claim 1 , wherein the measuring comprises measuring metrics relating to connectivity and processing performance of nodes comprising at least one server, network device, datacenter business offering, geographic location, and the distributed storage system as a whole, and connectivity and communication performance of communication links connecting the nodes.
4. The method of claim 3 , wherein the generated score for each node and link is further a function of scores of nodes and links lower in the hierarchy of the graph.
5. The method of claim 4 , wherein the generated score for each node and link is further determined according to:
Weight
level
n
=
∑
nodes
in
level
n
-
1
Weight
level
n
-
1
#
nodes
in
level
n
-
1
.
6. The method of claim 1 , wherein generating a list of additional failing components comprises:
removing the sub-tree of the failing component from the graph;
traversing the remaining nodes of the plurality of hierarchical levels of the graph, starting at a highest hierarchical level;
determining a node having a score that is greater than or equal to scores of any nodes in a sub-tree of the node at any lower level;
determining that the determined node is a failing component at that hierarchical level of the graph; and
repeating until no nodes having scores indicating problems remain in the graph.
7. A system comprising a processor, memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor to perform:
measuring problems and health of a plurality of physical and logical components in a distributed storage system, the plurality of physical and logical components forming nodes of the distributed storage system, and generating a graph of the nodes organized in a plurality of hierarchical levels;
generating, for each node in the graph, a score summarizing the measured problems and health of the node;
generating, for each node in the graph, a score summarizing the measured problems and health of the node;
determining a highest score at a highest hierarchical level of the graph by:
traversing the nodes of the plurality of hierarchical levels of the graph, starting at a highest hierarchical level,
determining a node having a score that is greater than or equal to scores of any nodes in a sub-graph of the node at any lower level, and
determining that the determined node is a failing component at a most significant level; and
determining an associated node as a failing component at a most significant level.
8. The system of claim 7 , further performing generating a list of additional failing components, the list ordered by a highest score of each successive node at each successive hierarchical level of the graph.
9. The system of claim 8 , wherein the generated score for each node and link is further a function of scores of nodes and links lower in the hierarchy of the graph.
10. The system of claim 7 , wherein the measuring comprises measuring metrics relating to connectivity and processing performance of nodes comprising at least one server, network device, datacenter business offering, geographic location, and the distributed storage system as a whole, and connectivity and communication performance of communication links connecting the nodes.
11. The system of claim 10 , wherein the generated score for each node and link is further determined according to:
Weight
level
n
=
∑
nodes
in
level
n
-
1
Weight
level
n
-
1
#
nodes
in
level
n
-
1
.
12. The system of claim 7 , wherein generating a list of additional failing components comprises:
removing the sub-tree of the failing component from the graph;
traversing the remaining nodes of the plurality of hierarchical levels of the graph, starting at a highest hierarchical level;
determining a node having a score that is greater than or equal to scores of any nodes in a sub-tree of the node at any lower level;
determining that the determined node is a failing component at that hierarchical level of the graph; and
repeating until no nodes having scores indicating problems remain in the graph.
13. A computer program product comprising a non-transitory computer readable storage having program instructions embodied therewith, the program instructions executable by a computer, to cause the computer to perform a method comprising:
measuring problems and health of a plurality of physical and logical components in a distributed storage system, the plurality of physical and logical components forming nodes of the distributed storage system, and generating a graph of the nodes organized in a plurality of hierarchical levels;
generating, for each node in the graph, a score summarizing the measured problems and health of the node;
generating, for each node in the graph, a score summarizing the measured problems and health of the node;
determining a highest score at a highest hierarchical level of the graph by:
traversing the nodes of the plurality of hierarchical levels of the graph, starting at a highest hierarchical level,
determining a node having a score that is greater than or equal to scores of any nodes in a sub-graph of the node at any lower level, and
determining that the determined node is a failing component at a most significant level; and
determining an associated node as a failing component at a most significant level.
14. The computer program product of claim 13 , further comprising generating a list of additional failing components, the list ordered by a highest score of each successive node at each successive hierarchical level of the graph.
15. The computer program product of claim 13 , wherein the measuring comprises measuring metrics relating to connectivity and processing performance of nodes comprising at least one server, network device, datacenter business offering, geographic location, and the distributed storage system as a whole, and connectivity and communication performance of communication links connecting the nodes.
16. The computer program product of claim 15 , wherein the generated score for each node and link is further a function of scores of nodes and links lower in the hierarchy of the graph, wherein the generated score for each node and link is further determined according to:
Weight
level
n
=
∑
nodes
in
level
n
-
1
Weight
level
n
-
1
#
nodes
in
level
n
-
1
.
17. The computer program product of claim 13 , wherein generating a list of additional failing components comprises:
removing the sub-tree of the failing component from the graph;
traversing the remaining nodes of the plurality of hierarchical levels of the graph, starting at a highest hierarchical level;
determining a node having a score that is greater than or equal to scores of any nodes in a sub-tree of the node at any lower level;
determining that the determined node is a failing component at that hierarchical level of the graph; and
repeating until no nodes having scores indicating problems remain in the graph.