Granted Patent B2

US 11,403,175 · App. 16/944,395 · Granted Aug 2, 2022

Inventors: Philippe Armangau (Acton, MA); Geng Han (Beijing, CN); Jian Gao (Beijing, CN)

Assignee: EMC IP Holding Company LLC

G06F11/1092G06F3/061G06F3/064G06F3/0607G06F3/0646G06F3/0689

Abstract

Claims (55)

1. A method comprising:

in response to detecting failure of a data storage drive, performing a drive rebuild operation by periodically:

calculating a target rebuild rate that enables data previously stored on the failed data storage drive to be completely rebuilt on at least one other data storage drive within an expected drive rebuild time window;

calculating a new value for at least one drive rebuild parameter based on a value for the at least one drive rebuild parameter corresponding to the target rebuild rate indicated by a static rebuild rate data structure and a current level of host I/O (Input/Output) activity; and

rebuilding the data previously stored on the failed data storage drive on a replacement data storage drive for a predetermined time period with the at least one drive rebuild parameter set to the new value for the drive rebuild parameter.

2. The method of claim 1 , wherein rebuilding the data previously stored on the failed data storage drive on the replacement data storage drive during the predetermined time period includes executing a plurality of rebuild jobs in parallel during the predetermined time period; and

wherein the at least one drive rebuild parameter includes a total number of processor cores on which corresponding individual drive rebuild jobs are scheduled for execution in parallel.

3. The method of claim 2 , further comprising:

determining the current level of host I/O activity at least in part by detecting a number of processor cores that are not currently performing host I/O processing; and

wherein calculating the new value for the total number of processor cores on which corresponding individual drive rebuild jobs are scheduled for execution in parallel during the predetermined time period includes:

comparing the number of processor cores not currently performing host I/O processing to the value for the total number of processor cores on which corresponding individual drive rebuild jobs are scheduled for execution in parallel indicated by the static rebuild rate data structure, and

in response to the number of processor cores not currently performing host I/O processing exceeding the total number of processor cores on which corresponding individual drive rebuild jobs are scheduled for execution in parallel indicated by the static rebuild rate data structure, setting the total number of processor cores on which corresponding individual drive rebuild jobs are scheduled for execution in parallel to the number of processor cores not currently performing host I/O processing.

4. The method of claim 3 , further comprising executing each of the individual drive rebuild jobs on the corresponding processor cores at a higher execution priority than the host I/O processing.

5. The method of claim 3 , further comprising:

assigning drive rebuild jobs to corresponding processor cores within a set of processor cores according to a first ordering of the processor cores within the set of processor cores; and

assigning host I/O processing to processor cores within the set of processor cores according to a second ordering of the processor cores within the set of processor cores, wherein the second ordering of the processor cores within the set of processor cores comprises a reverse order with respect to the first ordering of the processor cores within the set of processor cores.

6. The method of claim 2 , wherein rebuilding the data previously stored on the failed data storage drive further includes rebuilding data previously stored in multiple physical blocks in parallel; and

wherein the at least one drive rebuild parameter further includes a total number of physical blocks for which data is rebuilt in parallel by each one of the rebuild jobs during the predetermined time period.

7. The method of claim 2 , wherein the data previously stored on the failed data storage drive was protected using mapped RAID (Redundant Array of Independent Disks) data protection provided in a plurality of RAID extents; and

wherein each one of the drive rebuild jobs rebuilds data previously stored in a single one of the RAID extents.

8. The method of claim 1 , wherein the static rebuild rate data structure comprises a table of data elements storing a plurality of rebuild rates, and further comprising:

determining a data element in the table that stores a value equal to the target rebuild rate; and

determining the value for the at least one drive rebuild parameter corresponding to the target rebuild rate indicated by the static rebuild rate data structure as an index of the data element in the table that stores the value equal to the target rebuild rate.

9. The method of claim 1 , wherein the calculating of the target rebuild rate that enables the data previously stored on the failed data storage drive to be completely rebuilt on the replacement data storage drive within the expected rebuild time window is based at least in part on a percentage of the data previously stored on the failed data storage drive that has been rebuilt on the at least one other data storage drive and an amount of time that has elapsed since the drive rebuild operation began.

10. A data storage system comprising:

at least one storage processor including processing circuitry and a memory;

a plurality of non-volatile data storage drives communicably coupled to the storage processor; and

wherein the memory has program code stored thereon, wherein the program code, when executed by the processing circuitry, causes the processing circuitry to:

perform a drive rebuild operation by causing the processing circuitry to periodically:

calculate a target rebuild rate that enables data previously stored on the failed data storage drive to be completely rebuilt on at least one other data storage drive within an expected drive rebuild time window;

calculate a new value for at least one drive rebuild parameter based on a value for the at least one drive rebuild parameter corresponding to the target rebuild rate indicated by a static rebuild rate data structure and a current level of host I/O (Input/Output) activity; and

rebuild the data previously stored on the failed data storage drive on a replacement data storage drive for a predetermined time period with the at least one drive rebuild parameter set to the new value for the drive rebuild parameter.

11. The data storage system of claim 10 , wherein the program code, when executed, further causes the processing circuitry to rebuild the data previously stored on the failed data storage drive on the replacement data storage drive during the predetermined time period at least in part by executing a plurality of rebuild jobs in parallel during the predetermined time period; and

wherein the at least one drive rebuild parameter includes a total number of processor cores on which corresponding individual drive rebuild jobs are scheduled for execution in parallel.

12. The data storage system of claim 11 , wherein the program code, when executed, further causes the processing circuitry to:

determine the current level of host I/O activity at least in part by detecting a number of processor cores that are not currently performing host I/O processing; and

wherein the program code, when executed, causes the new value for the total number of processor cores on which corresponding individual drive rebuild jobs are scheduled for execution in parallel during the predetermined time period to be calculated at least in part by causing the processing circuitry to:

compare the number of processor cores not currently performing host I/O processing to the value for the total number of processor cores on which corresponding individual drive rebuild jobs are scheduled for execution in parallel indicated by the static rebuild rate data structure, and

in response to the number of processor cores not currently performing host I/O processing exceeding the total number of processor cores on which corresponding individual drive rebuild jobs are scheduled for execution in parallel indicated by the static rebuild rate data structure, set the total number of processor cores on which corresponding individual drive rebuild jobs are scheduled for execution in parallel to the number of processor cores not currently performing host I/O processing.

13. The data storage system of claim 12 , wherein the program code, when executed, further causes the processing circuitry to execute each of the individual drive rebuild jobs on the corresponding processor cores at a higher execution priority than the host I/O processing.

14. The data storage system of claim 12 , wherein the program code, when executed, further causes the processing circuitry to:

assign drive rebuild jobs to corresponding processor cores within a set of processor cores according to a first ordering of the processor cores within the set of processor cores; and

assign host I/O processing to processor cores within the set of processor cores according to a second ordering of the processor cores within the set of processor cores, wherein the second ordering of the processor cores within the set of processor cores comprises a reverse order with respect to the first ordering of the processor cores within the set of processor cores.

15. The data storage system of claim 11 , wherein the program code, when executed, causes the processing circuitry to rebuild the data previously stored on the failed data storage drive at least in part by causing the processing circuitry to rebuild data previously stored in multiple physical blocks in parallel; and

wherein the at least one drive rebuild parameter further includes a total number of physical blocks for which data is rebuilt in parallel by each one of the rebuild jobs during the predetermined time period.

16. The data storage system of claim 11 , wherein the data previously stored on the failed data storage drive was protected using mapped RAID (Redundant Array of Independent Disks) data protection provided in a plurality of RAID extents; and

wherein each one of the drive rebuild jobs rebuilds data previously stored in a single one of the RAID extents.

17. The data storage system of claim 10 , wherein the static rebuild rate data structure comprises a table of data elements storing a plurality of rebuild rates, and wherein the program code, when executed, further causes the processing circuitry to:

determine a data element in the table that stores a value equal to the target rebuild rate; and

determine the value for the at least one drive rebuild parameter corresponding to the target rebuild rate indicated by the static rebuild rate data structure as an index of the data element in the table that stores the value equal to the target rebuild rate.

18. The data storage system of claim 10 , wherein the calculation of the target rebuild rate that enables the data previously stored on the failed data storage drive to be completely rebuilt on the replacement data storage drive within the expected rebuild time window is based at least in part on a percentage of the data previously stored on the failed data storage drive that has been rebuilt on the at least one other data storage drive and an amount of time that has elapsed since the drive rebuild operation began.

19. A computer program product including a non-transitory computer readable medium having instructions stored thereon, wherein the instructions, when executed on processing circuitry, cause the processing circuitry to perform steps including:

calculating a target rebuild rate that enables data previously stored on the failed data storage drive to be completely rebuilt on at least one other data storage drive within an expected drive rebuild time window;

calculating a new value for at least one drive rebuild parameter based on a value for the at least one drive rebuild parameter corresponding to the target rebuild rate indicated by a static rebuild rate data structure and a current level of host I/O (Input/Output) activity; and

rebuilding the data previously stored on the failed data storage drive on a replacement data storage drive for a predetermined time period with the at least one drive rebuild parameter set to the new value for the drive rebuild parameter.

Assignments (9)

Continuity (1)