System and method for variable sparing in RAID groups based on drive failure probability
Drives of a RAID group are classified as either healthy or failing using a trained learning process. The failure probability is then determined for each failing drive using a Venn-ABERS framework which provides a boundary range on the failure prediction probability. A variable sparing mechanism is used to enable one or more drives of the RAID group to be used as dual-purpose drives. In a first state, the dual-purpose drives are user-addressable drives and are available to be used to process IO workload on the RAID group. Spreading the IO workload on the RAID group across a larger number of drives results in increased performance in the form of reduced latency. In a second state, the dual-purpose drives are not user-addressable and are spare drives in the RAID group, which improves the level of protection provided to data stored in the RAID group.
1. A non-transitory tangible computer readable storage medium having stored thereon a computer program for variable sparing in a RAID group based on drive failure probability, the RAID group having a set of capacity drives, one or more dual-purpose drives, and one or more spare drives, the computer program including a set of instructions which, when executed by a computer, cause the computer to perform a method comprising the steps of:
classifying each of the capacity drives of the RAID group as either healthy or being in danger of failing;
for each capacity drive classified as being in danger of failing, determining a respective failure probability boundary value for the capacity drive classified as being in danger of failing;
determining a number of failing drives of the RAID group, by comparing the failure probability boundary values of the capacity drives classified as being in danger of failing with a failure threshold;
comparing the number of failing drives with a minimum spare capacity value for the RAID group; and
when the number of failing drives is less than the minimum spare capacity value for the RAID group, setting a state of one of the dual-purpose drives of the RAID group to a first value to cause the dual-purpose drive to operate as a capacity drive in the RAID group such that IO workload on the RAID group is distributed across the capacity drives of the RAID group that are classified as healthy, the capacity drives of the RAID group that are classified as being in danger of failing, and the one of the dual-purpose drives; and
when the number of failing drives is greater than the minimum spare capacity value for the RAID group, setting the state of the one of the dual-purpose drives of the RAID group to a second value to cause the dual-purpose drive to operate as a dedicated spare drive in the RAID group such that the IO workload on the RAID group is distributed across the capacity drives of the RAID group that are classified as healthy and the capacity drives of the RAID group that are classified as being in danger of failing, and not distributed to the one of the dual-purpose drive until one of the healthy capacity drives of the RAID group, or one of the capacity drives of the RAID group classified as being in danger of failing, actually fails.
2. The non-transitory tangible computer readable storage medium of claim 1 , wherein the step of classifying each of a set of capacity drives of a RAID group as either being in danger of failing or healthy is implemented using a trained support vector machine classifier learning process.
3. The non-transitory tangible computer readable storage medium of claim 2 , further comprising providing, as feedback to the support vector machine classifier learning process, information related to drive failure events.
4. The non-transitory tangible computer readable storage medium of claim 1 , wherein the step of determining the failure probability boundary values for the capacity drives classified as being in danger of failing is implemented using a Venn-ABERS framework.
5. The non-transitory tangible computer readable storage medium of claim 1 , wherein setting the state to the first value to cause the dual-purpose drive to operate as a capacity drive in the RAID group further comprises a step of maintaining a capacity rule for the RAID group.
6. The non-transitory tangible computer readable storage medium of claim 5 , wherein the capacity rule for the RAID group includes a hard rule that an amount of data stored in the dual-purpose drive cannot exceed a sum of remaining capacity in the set of capacity drives of the RAID group.
7. The non-transitory tangible computer readable storage medium of claim 1 , wherein the step of setting the state of the one of the dual-purpose drives of the RAID group to a second value comprises copying data contained on the dual-purpose drive to the capacity drives of the RAID group.
8. A method of variable sparing in a RAID group based on drive failure probability, the RAID group having a set of capacity drives, one or more dual-purpose drives, and one or more spare drives, the method comprising the steps of:
classifying each of the capacity drives of the RAID group as either healthy or being in danger of failing;
for each capacity drive classified as being in danger of failing, determining a respective failure probability boundary value for the capacity drive classified as being in danger of failing;
determining a number of failing drives of the RAID group, by comparing the failure probability boundary values of the capacity drives classified as being in danger of failing with a failure threshold;
comparing the number of failing drives with a minimum spare capacity value for the RAID group; and
when the number of failing drives is less than the minimum spare capacity value for the RAID group, setting a state of one of the dual-purpose drives of the RAID group to a first value to cause the dual-purpose drive to operate as a capacity drive in the RAID group such that IO workload on the RAID group is distributed across the capacity drives of the RAID group that are classified as healthy, the capacity drives of the RAID group that are classified as being in danger of failing, and the one of the dual-purpose drives; and
when the number of failing drives is greater than the minimum spare capacity value for the RAID group, setting the state of the one of the dual-purpose drives of the RAID group to a second value to cause the dual-purpose drive to operate as a dedicated spare drive in the RAID group such that the IO workload on the RAID group is distributed across the capacity drives of the RAID group that are classified as healthy and the capacity drives of the RAID group that are classified as being in danger of failing, and not distributed to the one of the dual-purpose drive until one of the healthy capacity drives of the RAID group, or one of the capacity drives of the RAID group classified as being in danger of failing, actually fails.
9. The method of claim 8 , wherein the step of classifying each of a set of capacity drives of a RAID group as either being in danger of failing or healthy is implemented using a trained support vector machine classifier learning process.
10. The method of claim 9 , further comprising providing, as feedback to the support vector machine classifier learning process, information related to drive failure events.
11. The method of claim 8 , wherein the step of determining the failure probability boundary values for the capacity drives classified as being in danger of failing is implemented using a Venn-ABERS framework.
12. The method of claim 8 , wherein setting the state to the first value to cause the dual-purpose drive to operate as a capacity drive in the RAID group further comprises a step of maintaining a capacity rule for the RAID group.
13. The method of claim 12 , wherein the capacity rule for the RAID group includes a hard rule that an amount of data stored in the dual-purpose drive cannot exceed a sum of remaining capacity in the set of capacity drives of the RAID group.
14. The method of claim 8 , wherein the step of setting the state of the one of the dual-purpose drives of the RAID group to a second value comprises copying data contained on the selected dual-purpose drive to the capacity drives of the RAID group.
15. A storage system, comprising:
a RAID group including a set of capacity drives, a dual-purpose drive, and a spare drive, the dual-purpose drive being configured to operate either as a spare drive in the RAID group if user-addressability has not been enabled on the dual-purpose drive, or to operate as a capacity drive in the RAID group if user addressability has been enabled on the dual-purpose drive;
an analytics engine;
a RAID controller; and
a load balancer,
wherein the analytics engine contains control logic configured to:
classify each of the capacity drives of the RAID group as either healthy or being in danger of failing;
for each capacity drive classified as being in danger of failing, determine a respective failure probability boundary value for the capacity drive classified as being in danger of failing;
determine a number of failing drives of the RAID group, by comparing the failure probability boundary values of the capacity drives classified as being in danger of failing with a failure threshold;
compare the number of failing drives with a minimum spare capacity value for the RAID group; and
when the number of failing drives is less than the minimum spare capacity value for the RAID group, instruct the RAID controller to set a state of the dual-purpose drive of the RAID group to a first value to cause the dual-purpose drive to operate as a capacity drive in the RAID group such that IO workload on the RAID group is distributed across the capacity drives of the RAID group that are classified as healthy, the capacity drives of the RAID group that are classified as being in danger of failing, and the one of the dual-purpose drives; and
when the number of failing drives is greater than the minimum spare capacity value for the RAID group, set the state of the one of the dual-purpose drives of the RAID group to a second value to cause the dual-purpose drive to operate as a dedicated spare in the RAID group such that the IO workload on the RAID group is distributed across the capacity drives of the RAID group that are classified as healthy and the capacity drives of the RAID group that are classified as being in danger of failing, and not distributed to the one of the dual-purpose drive until one of the healthy capacity drives of the RAID group, or one of the capacity drives of the RAID group classified as being in danger of failing, actually fails.
16. The storage system of claim 15 , wherein the load balancer is configured to distribute the IO workload on the RAID group across the set of capacity drives when the dual-purpose drive is not user-addressable, and is configured to distribute the IO workload on the RAID group across the set of capacity drives and the dual-purpose drive when the user-addressability of the dual-purpose drive has been enabled.
17. The storage system of claim 15 , further comprising an addressability enabling latch for the dual-purpose node, and wherein the RAID controller uses the addressability enabling latch to set the state of the dual-purpose drive.
18. The storage system of claim 15 , wherein the load balancer is configured to maintain a capacity rule for the RAID group, the capacity rule for the RAID group includes a hard rule that an amount of data stored in the dual-purpose drive cannot exceed a sum of available capacity of each of the other drives of the RAID group.