Use of error correction-based metric for identifying poorly performing data storage devices
An approach to identifying poorly performing data storage devices (DSDs) in a data storage system, such as hard disk drives (HDDs) and/or solid-state drives (SSDs), involves retrieving and evaluating a respective set of log pages, such as SCSI Log Sense counters, from each of multiple DSDs. Based on each respective set of log pages, a value for a Quality of Service (QoS) metric is determined for each respective DSD, where each QoS value represents an average percentage of bytes processed without the respective DSD performing an autonomous error correction. In response to a particular DSD reaching a predetermined threshold QoS value, an in-situ repair may be determined for the particular DSD or the particular DSD may be added to a list of candidate DSDs for further examination, which may include an FRPH examination for suitably configured DSDs.
1. A method for identifying a poorly performing data storage device (DSD), the method comprising:
retrieving a respective set of error log pages from each of a plurality of data storage devices (DSDs);
based on each respective set of error log pages, determining a value for a Quality of Service (QoS) metric for each respective DSD of the plurality of DSDs, wherein each QoS value represents a percentage of total bytes processed without the respective DSD performing an autonomous error correction;
determining whether each DSD QoS value reaches a predetermined QoS threshold value; and
in response to a particular DSD QoS value reaching the predetermined QoS threshold value, determining whether to perform an in-situ repair for the particular DSD or to add the particular DSD to a list of candidate DSDs for further examination.
2. The method of claim 1 , wherein:
retrieving the set of error log pages comprises retrieving a set of SCSI (Small Computer System Interface) Log Sense counters; and
determining the value for the QoS metric comprises determining based on at least two parameters from a group of parameters consisting of (1) Parameter 0001h: errors corrected with possible delays, (2) Parameter 0004h: total times correction algorithm processed, and (3) Parameter 0005h: total bytes processed.
3. The method of claim 2 , wherein determining the value for the QoS metric comprises determining based on one domain from a group of domains consisting of read, write, and verify operational domains.
4. The method of claim 1 , wherein retrieving the set of error log pages comprises retrieving a set of SMART (Self-Monitoring, Analysis and Reporting Technology) parameters indicating (i) a number of total errors corrected, (ii) a number of correction algorithm invocations, and (iii) a number of total bytes processed.
5. The method of claim 1 , wherein determining the value for the QoS metric comprises determining the value for the QoS metric corresponding to a predetermined period of time less than a lifetime of each respective DSD.
6. The method of claim 1 , wherein determining whether a particular DSD QoS value reaches a predetermined QoS threshold value comprises comparing the particular DSD QoS value to a statistical measure of a QoS value corresponding to the plurality of DSDs.
7. The method of claim 6 , wherein the plurality of DSDs are affiliated in a RAID (Redundant Array of Independent Disks, or Drives) grouping of DSDs.
8. The method of claim 6 , wherein the plurality of DSDs are affiliated in an erasure coding paradigm applied to the plurality of DSDs.
9. The method of claim 1 , wherein determining whether to add the particular DSD to the list comprises determining whether to add the particular DSD to a list of candidate DSDs for further Full Recoveries Per Hour (FRPH) examination comprising:
retrieving a recovery log from the particular DSD which is a particular hard disk drive (HDD); and
based on the recovery log, determining a value for a FRPH metric for each respective read-write head of the particular HDD, including:
summing an amount of time the head spends on all recovery steps performed by the head over a particular length of time, including characterizing a full recovery as corresponding to a predetermined length of time the head spends on one or more recovery steps,
determining an equivalent number of full recoveries over the particular length of time, and
based on the number of full recoveries and the particular length of time, determining the value for the FRPH metric for the head.
10. The method of claim 9 , further comprising:
in response to a value for a particular head FRPH metric reaching a predetermined FRPH threshold value, determining an in-situ repair for the particular HDD in which the particular head operates.
11. A method for identifying a poorly performing data storage device (DSD) from a plurality of affiliated DSDs constituent to a data storage system, the method comprising:
periodically polling a set of error counters for the plurality of DSDs;
based on the error counters, determining a value for a lifetime Quality of Service (QoS) metric for each respective DSD of a plurality of DSDs, wherein each lifetime QoS value is based on an average percentage of total bytes processed over a lifetime of each respective DSD without the respective DSD performing an autonomous error correction; and
recording each lifetime QoS value for the polling period.
12. The method of claim 11 , further comprising:
identifying a particular percentage subset of the plurality of DSDs that exhibited a lowest lifetime QoS value during a most recent polling period;
for each DSD of the subset of DSDs whose most recent lifetime QoS value is more than one unit lower than that of a DSD having a highest lifetime QoS value within the subset, adding each such DSD to a list of candidate DSDs;
for each DSD of the subset of DSDs whose most recent lifetime QoS value has declined by more than one unit since computation of a particular prior lifetime QoS value, adding each such DSD to the list of candidate DSDs; and
for each DSD on the list of candidate DSDs that is not configured for computation of a corresponding Full Recoveries Per Hour (FRPH) value, determining whether to repair or retire each such DSD.
13. The method of claim 12 , further comprising:
for each DSD on the list of candidate DSDs that is configured for computation of the FRPH value, removing from the list of candidate DSDs any DSD that has passed a FRPH examination within a predetermined period of time less than the polling period; and
for each DSD remaining on the list of candidate DSDs, computing the FRPH value for each respective read-write head of each such DSD.
14. The method of claim 11 , wherein the plurality of DSDs are constituent to a single data storage system.
15. The method of claim 11 , wherein the plurality of DSDs are constituent to a plurality of data storage systems.
16. A data storage system comprising:
a system controller circuitry comprising memory and one or more processors and embodying one or more sequences of instructions which, when executed by the one or more processors, cause performance of:
retrieving a respective set of error log pages from each of a plurality of data storage devices (DSDs);
based on each respective set of error log pages, determining a value for a Quality of Service (QoS) metric for each respective DSD of the plurality of DSDs, wherein each QoS value represents a percentage of total bytes processed without the respective DSD performing an autonomous error correction;
determining whether each DSD QoS value reaches a predetermined QoS threshold value; and
in response to a particular DSD QoS value reaching the predetermined QoS threshold value, determining whether to perform an in-situ repair for the particular DSD or to add the particular DSD to a list of candidate DSDs for further Full Recoveries Per Hour (FRPH) examination.
17. The data storage system of claim 16 , wherein:
retrieving the set of error log pages comprises retrieving a set of SCSI (Small Computer System Interface) Log Sense counters; and
determining the value for the QoS metric comprises determining based on at least two parameters from a group of parameters consisting of (1) Parameter 0001h: errors corrected with possible delays, (2) Parameter 0004h: total times correction algorithm processed, and (3) Parameter 0005h: total bytes processed.
18. The data storage system of claim 16 , wherein determining whether a particular DSD QoS value reaches a predetermined QoS threshold value comprises comparing the particular DSD QoS to a statistical measure of a QoS value corresponding to the plurality of DSDs.
19. The data storage system of claim 16 , wherein the one or more processors are not configured to directly process floating-point arithmetic operations.
20. A data storage system comprising:
a plurality of data storage devices (DSDs);
means for retrieving a respective set of SCSI (Small Computer System Interface) log pages from each of the plurality of data storage devices (DSDs);
means for determining, based on each respective set of SCSI log pages, a value for a Quality of Service (QoS) metric for each respective DSD of the plurality of DSDs, wherein each QoS value represents a percentage of total bytes processed without the respective DSD performing an autonomous error correction;
means for determining whether each DSD QoS value reaches a predetermined QoS threshold value; and
means for determining, in response to a particular DSD QoS value reaching the predetermined QoS threshold value, whether to perform an in-situ repair for the particular DSD or to add the particular DSD to a list of candidate DSDs for further Full Recoveries Per Hour (FRPH) examination.