Using path quarantining to identify and handle backend issues
A technique involves, in response to encountering a predefined number of consecutive I/O errors using a drive path to a storage drive, transitioning the drive path from online to quarantined to temporarily deny further I/O operations from being processed using the drive path. The technique further involves starting a quarantine timer that defines a quarantine time period. The technique further involves performing an update operation that updates the drive path. The update operation (i) changes the drive path from quarantined to removed to continue denying further I/O operations from being processed using the drive path when a removal notification is received before the quarantine time period expires, and (ii) changes the drive path from quarantined to back to online to allow further I/O operations to be processed using the drive path when a removal notification is not received before the quarantine time period expires.
1. In a storage processor, a method of controlling access to a storage drive using a quarantined state, the method comprising:
in response to encountering a predefined number of consecutive input/output (I/O) errors using the drive path to the storage drive, transitioning a current state of the drive path from online to quarantined to temporarily deny further I/O operations from being processed using the drive path;
in response to transitioning the current state from online to quarantined, starting a quarantine timer that defines a quarantine time period; and
after the quarantine timer is started, performing an update operation that updates the current state of the drive path, the update operation (i) changing the current state of the drive path from quarantined to removed to continue denying further I/O operations from being processed using the drive path when a removal notification is received for the drive path before the quarantine time period expires, and (ii) changing the current state from quarantined to back to online to allow further I/O operations to be processed using the drive path when a removal notification is not received for the drive path before the quarantine time period expires.
2. A method as in claim 1 wherein transitioning the current state of the drive path from online to quarantined includes:
setting, by data path circuitry of the storage processor, a current state entry of a control structure to quarantined, the data path circuitry being permitted to process I/O operations using the drive path when the current state entry of the control structure is set to online and the data path circuitry not being permitted to process I/O operations using the drive path when the current state entry of the control structure is not set to online.
3. A method as in claim 2 wherein starting the quarantine timer that defines the quarantine time period includes:
setting, by the data path circuitry of the storage processor, the quarantine timer to cover (i) a first amount of time for platform management circuitry of the storage processor to determine whether the drive path failed due to a failed storage drive or a failed link, (ii) a second amount of time for the platform management circuitry of the storage processor to provide a notification to system manager circuitry of the storage processor which communicates with both the data path circuitry of the storage processor and other data path circuitry of another storage processor, and (iii) a third amount of time for the system manager circuitry of the storage processor to provide the removal notification to the data path circuitry before the quarantine time period expires.
4. A method as in claim 3 wherein performing the update operation that updates the current state of the drive path includes:
updating, by the data path circuitry of the storage processor, the current state entry of the control structure based on whether the data path circuitry of the storage processor received the removal notification from the system manager circuitry of the storage processor within the quarantine time period.
5. A method as in claim 4 wherein the data path circuitry of the storage processor receives the removal notification from the system manager circuitry of the storage processor within the quarantine time period; and
wherein updating the current state entry of the control structure includes:
changing the current state entry of the control structure from quarantined to removed to prevent further I/O operations from being processed using the drive path.
6. A method as in claim 4 wherein the data path circuitry of the storage processor does not receive the removal notification from the system manager circuitry of the storage processor within the quarantine time period; and
wherein updating the current state entry of the control structure includes:
changing the current state entry of the control structure from quarantined back to online to enable further I/O operations to be processed using the drive path.
7. A method as in claim 6 , further comprising:
incrementing a quarantined tally of the control structure, the quarantined tally indicating a number of times in a row that the current state entry of the control structure has transitioned between the online state and the quarantined state.
8. A method as in claim 7 , further comprising:
in response to the quarantined tally reaching a predefined tally threshold that is greater than 1, providing an alert notification and maintaining the current state entry of the control structure set to quarantined.
9. A method as in claim 4 , further comprising:
generating a current count of consecutive I/O errors encountered using the drive path and comparing the current count of consecutive I/O errors to a consecutive I/O error threshold to determine whether the data path circuitry of the storage processor has encountered the predefined number of consecutive I/O errors using the drive path.
10. A method as in claim 9 , further comprising:
detecting an I/O error using another drive path to the storage drive, and
in response to detecting the I/O error using the other drive path to the storage drive, resetting the current count of consecutive I/O errors using the drive path.
11. Data storage equipment, comprising:
a set of storage processors, each storage processor including memory, and control circuitry coupled to the memory, the memory storing instructions which, when carried out by the control circuitry, cause the control circuitry to:
in response to encountering a predefined number of consecutive input/output (I/O) errors using a drive path to a storage drive, transition a current state of the drive path from online to quarantined to temporarily deny further I/O operations from being processed using the drive path,
in response to transitioning the current state from online to quarantined, start a quarantine timer that measures a quarantine time period, and
after the quarantine timer is started, perform an update operation that updates the current state of the drive path, the update operation (i) changing the current state from quarantined to removed to continue denying further I/O operations from being processed using the drive path when a removal notification is received for the drive path before the quarantine time period expires, and (ii) changing the current state from quarantined to back to online to allow further I/O operations to be processed using the drive path when a removal notification is not received for the drive path before the quarantine time period expires.
12. Data storage equipment as in claim 11 , further comprising: a plurality of storage drives which couples to the set of storage processors;
wherein the set of storage processors includes at least two storage processors, each storage processor having at least two drive paths to each storage drive of the plurality of storage drives.
13. Data storage equipment as in claim 12 , further comprising:
a set of host computer interfaces which couples to the set of storage processors;
wherein the set of storage processors is constructed and arranged to store host data within the plurality of storage drives on behalf of a set of host computers that communicate with the set of storage processors through the set of host computer interfaces.
14. A computer program product having a non-transitory computer readable medium which stores a set of instructions to control access to a storage drive using a quarantined state; the set of instructions, when carried out by storage processing circuitry, causing the storage processing circuitry to perform a method of:
in response to encountering a predefined number of consecutive input/output (I/O) errors using the drive path to the storage drive, transitioning a current state of the drive path from online to quarantined to temporarily deny further I/O operations from being processed using the drive path;
in response to transitioning the current state from online to quarantined, starting a quarantine timer that measures a quarantine time period; and
after the quarantine timer is started, performing an update operation that updates the current state of the drive path, the update operation (i) changing the current state from quarantined to removed to continue denying further I/O operations from being processed using the drive path when a removal notification is received for the drive path before the quarantine time period expires, and (ii) changing the current state from quarantined to back to online to allow further I/O operations to be processed using the drive path when a removal notification is not received for the drive path before the quarantine time period expires.