Information processing system, information processing system management method, and program thereof

1. An information processing system comprising:

a plurality of storage nodes including processors, memories, and storage devices; and

a network for connecting the plurality of storage nodes,

wherein a processor of at least one storage node performs the steps of:

obtaining storage node management information related to a configuration of the storage nodes;

obtaining first failure information related to a first failure occurred in the at least one storage node;

calculating a failure occurrence probability that a second failure occurs based on the storage node management information and a rebuild time of a configuration of the information processing system for the first failure information;

calculating an availability level using the rebuild time and the failure occurrence probability, the availability level is a probability of transition to a state where the information processing system is stopped due to a failure;

notifying a necessity of maintenance work based on the availability level; and

controlling an operation of the information processing system based on the availability level by

copying information from one storage node of the plurality of storage nodes to a third storage node of the plurality of storage nodes and upon completion of the copying transfers differential data from the one storage node to the third storage node, on a condition that the availability level is higher than a threshold, and

searching the plurality of storage nodes for a copy of data stored in the at least one storage node with the first failure and additional data, selecting a storage node that does not contain a copy of the data and the additional data and copies the data and the additional data, on a condition that the availability level is lower than the threshold.

2. The information processing system according to claim 1 , wherein the processor calculates the availability level by calculating the failure occurrence probability that the second failure will occur during rebuild operation after the first failure occurs.

3. The information processing system according to claim 2 , wherein the rebuild time includes a time for reallocating data stored in the storage node, in which the first failure occurs, to another storage node.

4. The information processing system according to claim 3 , wherein the rebuild time includes time for adding a storage node to which the data is reallocated.

5. The information processing system according to claim 1 ,

wherein each of the processors include:

a first control unit for controlling reading and writing for a storage device; and

a second control unit for clustering storage node in cooperation with other nodes while calculating the availability level.

6. The information processing system according to claim 1 ,

wherein a group of the plurality of storage nodes to which failure is linked is set up as a failure group, and

wherein, according to the failure occurrence probability used for the calculation of the availability level, failures will occur at the same time in a plurality of storage nodes within the failure group upon occurrence of a failure.

7. The information processing system according to claim 1 , wherein the availability level is calculated based on the failure occurrence probability that the second failure will occur, as well as an operational continuity of the information processing system when the second failure occurs.

8. The information processing system according to claim 7 , wherein the operational continuity is conditional on having a predetermined redundancy or having a predetermined I/O performance.

9. The information processing system according to claim 1 , wherein

on the condition that the availability level is higher than the threshold the processor determines that the next state is not a failure state, and

on the condition that the availability level is lower than the threshold the processor determines that the next state will be the failure state.

10. A management method of an information processing system including a plurality of storage nodes including processors, memories, and storage devices, as well as a network for connecting the storage nodes,

wherein a processor of at least one storage node performs the steps of:

obtaining storage node management information related to a configuration of the storage nodes;

obtaining first failure information related to a first failure occurred in the at least one storage node;

calculating a failure occurrence probability that a second failure occurs based on the storage node management information and a rebuild time of a configuration of the information processing system for the first failure information;

calculating an availability level using the rebuild time and the failure occurrence probability, the availability level is a probability of transition to a state where the information processing system is stopped due to a failure;

notifying a necessity of maintenance work based on the availability level; and

controlling an operation of the information processing system based on the availability level by

copying information from one storage node of the plurality of storage nodes to a third storage node of the plurality of storage nodes and upon completion of the copying transfers differential data from the one storage node to the third storage node, on a condition that the availability level is higher than a threshold, and

searching the plurality of storage nodes for a copy of data stored in the at least one storage node with the first failure and additional data, selecting a storage node that does not contain a copy of the data and the additional data and copies the data and the additional data, on a condition that the availability level is lower than the threshold.

11. The management method of the information processing system according to claim 10 , wherein the processor calculates the availability level by calculating the failure occurrence probability that the second failure will occur during the rebuild operation after the first failure occurs.

12. A non-transitory computer-readable medium storing a program for controlling an information processing system including at least one storage node having a processor, a memory, and a storage device, when executed by the processor causes the processor to:

obtain storage node management information related to a configuration of the storage nodes;

obtain first failure information related to a first failure occurred in the at least one storage node;

calculate a failure occurrence probability that a second failure occurs based on the storage node management information and a rebuild time of a configuration of the information processing system for the first failure information;

calculate an availability level using the rebuild time and the failure occurrence probability, the availability level is a probability of transition to a state where the information processing system is stopped due to a failure;

notify a necessity of maintenance work based on the availability level; and

control an operation of the information processing system based on the availability level by

copying information from one storage node of the plurality of storage nodes to a third storage node of the plurality of storage nodes and upon completion of the copying transfers differential data from the one storage node to the third storage node, on a condition that the availability level is higher than a threshold, and

searching the plurality of storage nodes for a copy of data stored in the at least one storage node with the first failure and additional data, selecting a storage node that does not contain a copy of the data and the additional data and copies the data and the additional data, on a condition that the availability level is lower than the threshold.