Use of recovery behavior for prognosticating and in-situ repair of data storage devices
An approach to identifying problematic data storage devices, such as hard disk drives (HDDs), in a data storage system involves retrieving and evaluating a respective recovery log, such as a media error section of a device status log, from each of multiple HDDs. Based on each recovery log, a value for a Full Recoveries Per Hour (FRPH) metric is determined for each read-write head of each respective HDD. Generally, the FRPH metric characterizes the amount of time a head has spent performing recovery operations. In response to a particular head FRPH reaching a pre-determined threshold value, an in-situ repair can be determined for the HDD in which the head operates. Similarly, in the context of solid-state drives (SSDs), a latency metric is determinable based on time spent waiting on resolving input/output (IO) request collisions, on which an in-situ repair can be based.
1. A method for identifying a problematic hard disk drive (HDD), the method comprising:
retrieving a respective recovery log from each of a plurality of hard disk drives (HDDs);
based on each respective recovery log, determining a value for a Full Recoveries Per Hour (FRPH) metric for each respective read-write head of each respective HDD of the plurality of HDDs, including:
summing an amount of time the head spends on all recovery steps performed by the head over a particular length of time for evaluation, including characterizing a full recovery as corresponding to a pre-determined length of time the head spends on one or more recovery steps,
determining an equivalent number of full recoveries over the particular length of time based on the amount of time summed, and
based on the number of full recoveries determined and the particular length of time, determining the FRPH value for the head; and
in response to a particular head FRPH value reaching a pre-determined FRPH threshold value, determining an in-situ repair for the HDD in which the particular head operates.
2. The method of claim 1 , wherein the plurality of HDDs are constituent to a single data storage system.
3. The method of claim 2 , further comprising:
determining the FRPH threshold value based on a mean or median value of FRPH for the plurality of HDDs constituent to the single data storage system.
4. The method of claim 1 , wherein the plurality of HDDs are constituent to a plurality of data storage systems.
5. The method of claim 4 , further comprising:
determining the FRPH threshold value based on a mean or median value of FRPH for the plurality of HDDs constituent to the plurality of data storage systems.
6. The method of claim 1 , wherein the particular length of time is a configurable parameter.
7. The method of claim 1 , wherein determining the value for FRPH further comprises identifying a problematic HDD as an HDD having one or more heads exhibiting an FRPH value equal to or greater than the FRPH threshold value of one full recovery over a 300-hour time period.
8. The method of claim 1 , wherein determining the in-situ repair for the HDD comprises:
identifying the HDD as a problematic HDD based on having one or more heads exhibiting an FRPH value equal to or greater than the FRPH threshold value over the particular length of time;
identifying the head, within the problematic HDD, having the highest FRPH value; and
classifying the head having the highest FRPH value as ineligible for use in further input/output (IO) operations.
9. The method of claim 8 , wherein determining the in-situ repair for the HDD comprises:
if no head exhibits an FRPH value equal to or greater than the FRPH threshold value over the particular length of time, then reformatting the HDD to its current storage capacity.
10. The method of claim 1 , wherein determining the in-situ repair for the HDD comprises:
identifying one or more troublesome regions of storage media contributing to the FRPH value; and
marking the one or more troublesome regions of storage media unavailable for further input/output (IO) operations.
11. A data storage system comprising:
a plurality of data storage devices; and
a system controller circuitry comprising memory and one or more processors and embodying one or more sequences of instructions which, when executed by the one or more processors, cause performance of:
retrieving a respective recovery log from each of a plurality of hard disk drives (HDDs);
based on each respective recovery log, determining a value for a metric Full Recoveries Per Hour (FRPH) metric for each respective read-write head of each respective HDD of the plurality of HDDs, including:
summing an amount of time the head spends on all recovery steps performed by the head over a particular length of time, including characterizing a full recovery as corresponding to a pre-determined length of time the head spends on one or more recovery steps,
determining an equivalent number of full recoveries over the particular length of time based on the amount of time summed, and
based on the number of full recoveries determined and the particular length of time, determining the FRPH value for the head; and
in response to a particular head FRPH value reaching a pre-determined FRPH threshold value, determining an in-situ repair for the HDD in which the particular head operates.
12. The data storage system of claim 11 , wherein the one or more sequences of instructions cause further performance of:
determining the FRPH threshold value based on a mean or median value of FRPH for the plurality of HDDs constituent to the data storage system.
13. The data storage system of claim 11 , wherein the one or more sequences of instructions cause further performance of:
determining the FRPH threshold value based on a mean or median value of FRPH for the plurality of HDDs constituent to a plurality of data storage systems.
14. The data storage system of claim 11 , wherein determining the in-situ repair for the HDD comprises:
identifying the HDD as a problematic HDD based on having one or more heads exhibiting an FRPH value equal to or greater than the FRPH threshold value over the particular length of time;
identifying the head, within the problematic HDD, having the highest FRPH value; and
classifying the head having the highest FRPH value as ineligible for use in further input/output (IO) operations.
15. The data storage system of claim 14 , wherein determining the in-situ repair for the HDD comprises:
if no head exhibits an FRPH value equal to or greater than the FRPH threshold value over the particular length of time, then reformatting the HDD to its current storage capacity.
16. The data storage system of claim 11 , wherein determining the in-situ repair for the HDD comprises:
identifying one or more troublesome regions of storage media contributing to the FRPH value; and
marking the one or more troublesome regions of storage media unavailable for further input/output (IO) operations.
17. A data storage system comprising:
a plurality of hard disk drives (HDDs);
means for retrieving a respective recovery log from each of the plurality of HDDs;
means for determining, based on each respective recovery log, a value for a metric Full Recoveries Per Hour (FRPH) metric for each respective read-write head of each respective HDD of the plurality of HDDs, including:
means for summing an amount of time the head spends on all recovery steps performed by the head over a particular length of time, including characterizing a full recovery as corresponding to a pre-determined length of time the head spends on one or more recovery steps,
means for determining an equivalent number of full recoveries over the particular length of time based on the amount of time summed, and
means for determining the FRPH value for the head based on the number of full recoveries determined and the particular length of time; and
means for determining, in response to a particular head FRPH value reaching a pre-determined FRPH threshold value, an in-situ repair for the HDD in which the particular head operates.
18. The data storage system of claim 17 , wherein the means for determining the in-situ repair for the HDD comprises:
means for identifying the HDD as a problematic HDD based on having one or more heads exhibiting an FRPH value equal to or greater than the FRPH threshold value over the particular length of time;
means for identifying the head, within the problematic HDD, having the highest FRPH value; and
means for classifying the head having the highest FRPH value as ineligible for use in further input/output (IO) operations.
19. The data storage system of claim 18 , wherein the means for determining the in-situ repair for the HDD comprises:
means for reformatting the HDD to its current storage capacity if no head exhibits an FRPH value equal to or greater than the FRPH threshold value over the particular length of time.
20. The data storage system of claim 17 , wherein the means for determining the in-situ repair for the HDD comprises:
means for identifying one or more troublesome regions of storage media contributing to the FRPH value; and
means for marking the one or more troublesome regions of storage media unavailable for further input/output (IO) operations.
21. A data storage system comprising:
a plurality of solid-state data storage devices (SSDs); and
a system controller circuitry comprising memory and one or more processors and embodying one or more sequences of instructions which, when executed by the one or more processors, cause performance of:
based on host input/output (IO) request collisions, determining a value for a waiting metric for each respective SSD of the plurality of SSDs, including:
summing an amount of wait time spent on collisions by the SSD over a particular length of time, including characterizing a full wait as corresponding to a pre-determined length of time the SSD spends resolving collisions,
determining an equivalent number of full waits over the particular length of time based on the amount of wait time summed, and
based on the number of full waits determined and the particular length of time, determining the waiting metric value for the SSD; and
in response to a particular SSD waiting metric value reaching a pre-determined threshold value, determining an in-situ repair for the SSD.