Storage system and storage node management method
A worker node included in a storage system 1 includes a score calculation unit 31 that calculates a score of the worker node based on a failure history and an operation status of the worker node, and a master node (P) includes a promotion node selection unit 52 that compares scores for each worker node when a failure occurs in one of master nodes and selects, based on the scores, a worker node to be promoted to a master node instead of the master node in which the failure has occurred.
1. A storage system in which a plurality of storage nodes is coupled by a network, each of the plurality of storage nodes including a processor, a memory, and a storage device, and a cluster is configured by using the plurality of storage nodes as master nodes and using remaining storage nodes as worker nodes, wherein
the processor of each of the worker nodes includes a score calculation unit that calculates a score of the worker node based on a failure history and an operation status of the worker node, and
the processor of each of the master nodes includes a promotion node selection unit that compares the scores for the worker nodes when a failure occurs in one of the master nodes, and selects, based on the scores, the worker node to be promoted to the master node instead of the master node in which the failure has occurred,
wherein the processor of the master node includes a master-side reconfiguration unit that promotes the worker node selected by the promotion node selection unit to the master node and reconfigures the cluster, and
wherein a plurality of the master nodes are configured with a certain degree of redundancy, and
the promotion node selection unit selects the worker node to be promoted to the master node when a number of the master nodes cannot ensure the redundancy.
2. The storage system according to claim 1 , wherein
the score calculation unit calculates the score as information with which the promotion node selection unit can compare the failure history and the operation status of the worker node for each index classification obtained by classifying an index, and
the memory of the master node has a score table in which score information is stored for each worker node and for each index classification.
3. The storage system according to claim 2 , wherein
the index classification includes at least one of a main body failure that is an index of the failure history of the worker node, a network failure that is an index of the failure history of the network to which the worker node is coupled, an operating time that is an index related to an operation of the worker node, and a virtualization status that is an index related to a status of a virtualization environment of the worker node.
4. The storage system according to claim 2 , wherein
the promotion node selection unit compares the scores for each index classification and for a plurality of the worker nodes according to a priority order defined in the index classification, and in a case where scores of the index classification having a higher priority order are equal in the plurality of worker nodes, the promotion node selection unit repeats processing of comparing the scores with the scores of the index classification having a next highest priority order, and selects the worker nodes with the same index classification and a higher score.
5. The storage system according to claim 2 , wherein
the processor of the master node includes a score management unit that manages the score information received from the worker node based on the score table and invalidates, from the score table, the score of the worker node from which the score information cannot be received for a predetermined time or more.
6. The storage system according to claim 1 , wherein
the master-side reconfiguration unit removes the master node in which a failure has occurred from the cluster and updates information on the storage node managed as the master node based on information on the worker node to be promoted to the master node, and
the processor of the worker node to be promoted to the master node includes a worker-side reconfiguration unit that is activated to operate as the master node according to an instruction of the promotion node selection unit and reconfigures information necessary as the master node in synchronization with the master-side reconfiguration unit.
7. The storage system according to claim 1 , wherein
a fault domain is set for each of the plurality of storage nodes sharing at least one of a power supply system and a network switch, and
the promotion node selection unit selects the worker node to be promoted to the master node in units of the fault domain.
8. The storage system according to claim 7 , wherein
the promotion node selection unit selects the worker node to be promoted to the master node from the worker nodes included in the fault domain having a smallest number of master nodes.
9. The storage system according to claim 1 , wherein
one of the plurality of master nodes is configured as a primary used as an active system, and rest are configured as a secondary used as a standby system, and
the master node configured as the primary receives the score information from the worker node, and reflects the score information in the master node configured as the secondary.
10. A storage node management method performed in a storage system in which a plurality of storage nodes is coupled by a network, each of the plurality of storage nodes including a processor, a memory, and a storage device, and a cluster is configured by using the plurality of storage nodes as master nodes and using remaining storage nodes as worker nodes, the storage node management method comprising:
calculating, by the processor of each of the worker nodes, a score of the worker node based on a failure history and an operation status of the worker node; and
comparing, by the processor of each of the master nodes, the scores for the worker nodes when a failure occurs in one of the master nodes, and selecting, based on the scores, the worker node to be promoted to the master node instead of the master node in which the failure has occurred,
wherein the processor of the master node includes a master-side reconfiguration unit that promotes the worker node selected by the promotion node selection unit to the master node and reconfigures the cluster, and
wherein a plurality of the master nodes are configured with a certain degree of redundancy, and
the promotion node selection unit selects the worker node to be promoted to the master node when a number of the master nodes cannot ensure the redundancy.