Reducing concurrency bottlenecks while rebuilding a failed drive in a data storage system
A concurrent rebuild list indicates RAID extents to be concurrently rebuilt in response to a physical drive failure. When rebuilding of a RAID extent in the list completes, a next RAID extent to add to the list is selected that has a lowest relatedness score in a candidate set of RAID extents. The relatedness score indicates an amount of limitation with regard to concurrently rebuilding the candidate RAID extent in combination with the RAID extents remaining in the concurrent rebuild list. The relatedness score may be a sum of a weighted write score indicating limits on concurrent write operations when rebuilding a candidate RAID extent in combination with the RAID extents remaining in the concurrent rebuild list, and a read score indicating limits on concurrent read operations when rebuilding the candidate RAID extent in combination with the RAID extents remaining in the concurrent rebuild list.
1. A method of providing RAID (Redundant Array of Independent Disks) data protection for a storage object in a data storage system, wherein the data storage system includes a storage processor and a set of physical drives communicably coupled to the storage processor, the method comprising:
generating a RAID mapping table, wherein the RAID mapping table contains a plurality of RAID extents, wherein each RAID extent contained in the RAID mapping table indicates a plurality of drive extents for storing host data written to the storage object and related parity information, and wherein each drive extent comprises a contiguous region of non-volatile data storage in one of the physical drives;
in response to detecting that one of the physical drives has failed, concurrently rebuilding RAID extents in a concurrent rebuild list, wherein each RAID extent in the concurrent rebuild list indicates a drive extent of the failed one of the physical drives, and wherein for each one of the RAID extents in the concurrent rebuild list rebuilding includes i) recovering host data previously stored in the drive extent of the failed one of the physical drives indicated by the RAID extent, and ii) writing the recovered host data to a spare drive extent allocated to the RAID extent;
in response to detecting that rebuilding of one of the RAID extents in the concurrent rebuild list has completed, removing that one of the RAID extents from the concurrent rebuild list, and selecting a next RAID extent to replace the removed RAID extent in the concurrent rebuild list by
i) forming a candidate set of RAID extents, wherein each RAID extent in the candidate set indicates a drive extent of the failed physical drive and has not been rebuilt,
ii) calculating a relatedness score for each RAID extent in the candidate set with respect to the RAID extents remaining in the concurrent rebuild list, wherein the relatedness score indicates an amount of limitation with regard to concurrently rebuilding the RAID extent in combination with the RAID extents remaining in the concurrent rebuild list, and
iii) selecting as the new RAID extent to replace the removed RAID extent in the concurrent rebuild list a RAID extent in the candidate set having a lowest relatedness score of the RAID extents in the candidate set.
2. The method of claim 1 , wherein calculating the relatedness score for each RAID extent in the candidate set further comprises:
creating a candidate concurrent rebuild list for the RAID extent made up of the RAID extent and the RAID extents remaining in the concurrent rebuild list;
calculating a write score for the RAID extent that is equal to a total number of pairs of spare drive extents in which both spare drive extents of the pair are both i) allocated to the RAID extents in the candidate concurrent rebuild list, and ii) located within a single one of the physical drives; and
wherein calculating the relatedness score for the RAID extent is based at least in part on the write score for the RAID extent.
3. The method of claim 2 , wherein calculating the relatedness score for each RAID extent in the candidate set further comprises:
calculating a read score for the RAID extent that is equal to a number of pairs of drive extents in which both drive extents of the pair are both i) indicated by the RAID extents in the candidate concurrent rebuild list, and ii) located within a single one of the physical drives; and
wherein calculating the relatedness score for the RAID extent comprises generating a sum of the write score for the RAID extent and the read score for the RAID extent.
4. The method of claim 3 , further comprising
prior to generating the sum of the write score for the RAID extent and the read score for the RAID extent, applying a predetermined write score weighting to the write score for the RAID extent, wherein the predetermined write score weighting causes the write score for the RAID extent to be more heavily weighted than the read score for the RAID extent in the relatedness score for the candidate RAID extent.
5. The method of claim 4 , wherein concurrently rebuilding the RAID extents in the concurrent rebuild list comprises concurrently reading drive extents indicated by different ones of the RAID extents in the concurrent rebuild list to recover host data previously stored on drive extents of the failed one of the physical drives.
6. The method of claim 5 , wherein concurrently rebuilding the RAID extents in the concurrent rebuild list comprises concurrently writing spare drive extents allocated to different ones of the RAID extents in the concurrent rebuild list with recovered host data previously stored on the drive extents of the failed one of the physical drives.
7. A data storage system that provides RAID (Redundant Array of Independent Disks) data protection for a storage object, comprising:
at least one storage processor including processing circuitry and a memory;
a set of physical drives communicably coupled to the storage processor; and
wherein the storage processor is configured and arranged to:
generate a RAID mapping table, wherein the RAID mapping table contains a plurality of RAID extents, wherein each RAID extent contained in the RAID mapping table indicates a plurality of drive extents for storing host data written to the storage object and related parity information, and wherein each drive extent comprises a contiguous region of non-volatile data storage in one of the physical drives;
in response to detecting that one of the physical drives has failed, concurrently rebuild RAID extents in a concurrent rebuild list, wherein each RAID extent in the concurrent rebuild list indicates a drive extent of the failed one of the physical drives, and wherein each one of the RAID extents in the concurrent rebuild list is rebuilt at least in part by i) recovering host data previously stored in the drive extent of the failed one of the physical drives indicated by the RAID extent, and ii) writing the recovered host data to a spare drive extent allocated to the RAID extent;
in response to detecting that rebuilding of one of the RAID extents in the concurrent rebuild list has completed, remove that one of the RAID extents from the concurrent rebuild list, and select a next RAID extent to replace the removed RAID extent in the concurrent rebuild list by operating to
i) form a candidate set of RAID extents, wherein each RAID extent in the candidate set indicates a drive extent of the failed physical drive and has not been rebuilt,
ii) calculate a relatedness score for each RAID extent in the candidate set with respect to the RAID extents remaining in the concurrent rebuild list, wherein the relatedness score indicates an amount of limitation with regard to concurrently rebuilding the RAID extent in combination with the RAID extents remaining in the concurrent rebuild list, and
iii) select as the new RAID extent to replace the removed RAID extent in the concurrent rebuild list a RAID extent in the candidate set having a lowest relatedness score of the RAID extents in the candidate set.
8. The data storage system of claim 7 , wherein to calculate the relatedness score for each RAID extent in the candidate set the storage processor is further configured and arranged to:
create a candidate concurrent rebuild list for the RAID extent made up of the RAID extent and the RAID extents remaining in the concurrent rebuild list;
calculate a write score for the RAID extent that is equal to a total number of pairs of spare drive extents in which both spare drive extents of the pair are both i) allocated to the RAID extents in the candidate concurrent rebuild list, and ii) located within a single one of the physical drives; and
calculate the relatedness score for the RAID extent at least in part based on the write score for the RAID extent.
9. The data storage system of claim 8 , wherein to calculate the relatedness score for each RAID extent in the candidate set, the storage processor is further configured and arranged to:
calculate a read score for the RAID extent that is equal to a number of pairs of drive extents in which both drive extents of the pair are both i) indicated by the RAID extents in the candidate concurrent rebuild list, and ii) located within a single one of the physical drives; and
calculate the relatedness score for the RAID extent by generating a sum of the write score for the RAID extent and the read score for the RAID extent.
10. The data storage system of claim 9 , wherein the storage processor is further configured and arranged to:
prior to generating the sum of the write score for the RAID extent and the read score for the RAID extent, apply a predetermined write score weighting to the write score for the RAID extent, wherein the predetermined write score weighting causes the write score for the RAID extent to be more heavily weighted than the read score for the RAID extent in the relatedness score for the candidate RAID extent.
11. The data storage system of claim 10 , wherein the storage processor is further configured and arranged to concurrently rebuild the RAID extents in the concurrent rebuild list by operating to concurrently read drive extents indicated by different ones of the RAID extents in the concurrent rebuild list to recover host data previously stored on drive extents of the failed one of the physical drives.
12. The data storage system of claim 11 , wherein the storage processor is further configured and arranged to concurrently rebuild the RAID extents in the concurrent rebuild list by operating to concurrently write spare drive extents allocated to different ones of the RAID extents in the concurrent rebuild list with recovered host data previously stored on the drive extents of the failed one of the physical drives.
13. A computer program product, comprising:
a non-transitory computer readable medium storing program code for providing RAID (Redundant Array of Independent Disks) data protection for a storage object in a data storage system, wherein the data storage system includes a storage processor and a set of non-volatile data storage devices communicably coupled to the storage processor, the set of instructions, when carried out by at least one processor in the storage processor, causing the storage processor to perform a method of:
generating a RAID mapping table, wherein the RAID mapping table contains a plurality of RAID extents, wherein each RAID extent contained in the RAID mapping table indicates a plurality of drive extents for storing host data written to the storage object and related parity information, and wherein each drive extent comprises a contiguous region of non-volatile data storage in one of the physical drives;
in response to detecting that one of the physical drives has failed, concurrently rebuilding RAID extents in a concurrent rebuild list, wherein each RAID extent in the concurrent rebuild list indicates a drive extent of the failed one of the physical drives, and wherein for each one of the RAID extents in the concurrent rebuild list rebuilding includes i) recovering host data previously stored in the drive extent of the failed one of the physical drives indicated by the RAID extent, and ii) writing the recovered host data to a spare drive extent allocated to the RAID extent;
in response to detecting that rebuilding of one of the RAID extents in the concurrent rebuild list has completed, removing that one of the RAID extents from the concurrent rebuild list, and selecting a next RAID extent to replace the removed RAID extent in the concurrent rebuild list by
i) forming a candidate set of RAID extents, wherein each RAID extent in the candidate set indicates a drive extent of the failed physical drive and has not been rebuilt,
ii) calculating a relatedness score for each RAID extent in the candidate set with respect to the RAID extents remaining in the concurrent rebuild list, wherein the relatedness score indicates an amount of limitation with regard to concurrently rebuilding the RAID extent in combination with the RAID extents remaining in the concurrent rebuild list, and
iii) selecting as the new RAID extent to replace the removed RAID extent in the concurrent rebuild list a RAID extent in the candidate set having a lowest relatedness score of the RAID extents in the candidate set.
14. The computer program product of claim 13 , wherein the method performed by the storage processor when the set of instructions is carried out by at least one processor in the storage processor further comprises calculating the relatedness score for each RAID extent in the candidate set at least in part by:
creating a candidate concurrent rebuild list for the RAID extent made up of the RAID extent and the RAID extents remaining in the concurrent rebuild list;
calculating a write score for the RAID extent that is equal to a total number of pairs of spare drive extents in which both spare drive extents of the pair are both i) allocated to the RAID extents in the candidate concurrent rebuild list, and ii) located within a single one of the physical drives; and
wherein calculating the relatedness score for the RAID extent is based at least in part on the write score for the RAID extent.
15. The computer program product of claim 14 , wherein the method performed by the storage processor when the set of instructions is carried out by at least one processor in the storage processor further comprises calculating the relatedness score for each RAID extent in the candidate set at least in part by:
calculating a read score for the RAID extent that is equal to a number of pairs of drive extents in which both drive extents of the pair are both i) indicated by the RAID extents in the candidate concurrent rebuild list, and ii) located within a single one of the physical drives; and
wherein calculating the relatedness score for the RAID extent comprises generating a sum of the write score for the RAID extent and the read score for the RAID extent.
16. The computer program product of claim 15 , wherein the method performed by the storage processor when the set of instructions is carried out by at least one processor in the storage processor further comprises:
prior to generating the sum of the write score for the RAID extent and the read score for the RAID extent, applying a predetermined write score weighting to the write score for the RAID extent, wherein the predetermined write score weighting causes the write score for the RAID extent to be more heavily weighted than the read score for the RAID extent in the relatedness score for the candidate RAID extent.
17. The computer program product of claim 16 , wherein the method performed by the storage processor when the set of instructions is carried out by at least one processor in the storage processor further comprises concurrently rebuilding the RAID extents in the concurrent rebuild list at least in part by concurrently reading drive extents indicated by different ones of the RAID extents in the concurrent rebuild list to recover host data previously stored on drive extents of the failed one of the physical drives.
18. The computer program product of claim 17 , wherein the method performed by the storage processor when the set of instructions is carried out by at least one processor in the storage processor further comprises concurrently rebuilding the RAID extents in the concurrent rebuild list at least in part by concurrently writing spare drive extents allocated to different ones of the RAID extents in the concurrent rebuild list with recovered host data previously stored on the drive extents of the failed one of the physical drives.