Lifecycle of handling faults in next generation storage systems
Techniques for providing lifecycle handling of faults associated with a storage appliance. The techniques can include recording information or details of one or more detected faults associated with a storage appliance in a fault database, analyzing the recorded fault information/details by an analytic tool, and determining whether to perform immediate recovery or defer recovery of the detected fault(s) based at least on a recommendation of the analytic tool. In this way, taking the storage appliance offline upon occurrence of a single fault can, for the most part, be avoided. In addition, because the detection of faults is not performed within a fault domain for a filesystem handling files for a single or limited number of volume families, a deduplication domain for the files of the filesystem is not restricted to the single or limited number of volume families.
1. A method of lifecycle handling of faults associated with a storage appliance, comprising:
recording and aggregating fault data pertaining to a plurality of detected faults associated with a storage appliance in a fault database;
analyzing the aggregated fault data to generate a recommendation for recovery of a detected fault from among the plurality of detected faults;
performing immediate or deferred recovery of the detected fault based at least on the recommendation, thereby reducing occurrences of the storage appliance being taken offline upon detection of a single fault;
determining that the detected fault is correctable; and
having determined that the detected fault is correctable, determining whether the recovery of the detected fault is deferrable or undeferrable,
wherein performing immediate recovery of the detected fault includes, having determined that the recovery of the detected fault is undeferrable, taking the storage appliance offline, and
wherein performing deferred recovery of the detected fault includes, having determined that the recovery of the detected fault is deferrable, scheduling the recovery of the detected fault to be performed at a later time.
2. The method of claim 1 further comprising:
having performed the immediate or deferred recovery of the detected fault, archiving and clearing the fault database, thereby readying the storage appliance for normal operation.
3. The method of claim 1 further comprising:
generating at least one fault alert notification containing the fault data.
4. The method of claim 3 further comprising:
sending, by a fault alert application programming interface (API), the at least one fault alert notification containing the fault data to a management control path of the storage appliance for transmission from the storage appliance to a storage administrator computer.
5. The method of claim 4 further comprising:
extracting at least some of the fault data from the at least one fault alert notification.
6. The method of claim 1 further comprising:
having determined that the recovery of the detected fault is undeferrable, scheduling the recovery of the detected fault to be performed immediately.
7. The method of claim 1 further comprising:
upon completion of the recovery of the detected fault, clearing or resetting a record of the detected fault from the fault database.
8. The method of claim 1 further comprising:
having determined that the recovery of the detected fault is deferrable, continuing to aggregate fault data and analyze the aggregated fault data for subsequent detected faults associated with the storage appliance.
9. The method of claim 8 further comprising:
determining whether or not the recovery of the detected fault is deferrable in view of the continued aggregation and analysis of the aggregated fault data for the subsequent detected faults associated with the storage appliance.
10. The method of claim 9 further comprising:
having determined that the recovery of the detected fault is not deferrable in view of the continued aggregation and analysis of the aggregated fault data, changing a state of the recovery of the detected fault from deferrable to undeferrable; and
scheduling the recovery of the detected fault to be performed immediately.
11. A system for lifecycle handling of faults associated with a storage appliance, comprising:
a memory; and
processing circuitry configured to execute program instructions out of the memory to:
record and aggregate fault data pertaining to a plurality of detected faults associated with a storage appliance in a fault database;
analyze the aggregated fault data to generate a recommendation for recovery of a detected fault from among the plurality of detected faults;
perform immediate or deferred recovery of the detected fault based at least on the recommendation, thereby reducing occurrences of the storage appliance being taken offline upon detection of a single fault;
determine that the detected fault is correctable; and
having determined that the detected fault is correctable, determine whether the recovery of the detected fault is deferrable or undeferrable,
wherein to perform immediate recovery of the detected fault includes, having determined that the recovery of the detected fault is undeferrable, to take the storage appliance offline, and
wherein to perform deferred recovery of the detected fault includes, having determined that the recovery of the detected fault is deferrable, to schedule the recovery of the detected fault to be performed at a later time.
12. The system of claim 11 wherein the processing circuitry is further configured to execute the program instructions out of the memory, having determined that the recovery of the detected fault is deferrable, to continue to aggregate fault data and analyze the aggregated fault data for subsequent detected faults associated with the storage appliance.
13. The system of claim 12 wherein the processing circuitry is further configured to execute the program instructions out of the memory to determine whether or not the recovery of the detected fault is deferrable in view of the continued aggregation and analysis of the aggregated fault data for the subsequent detected faults associated with the storage appliance.
14. The system of claim 13 wherein the processing circuitry is further configured to execute the program instructions out of the memory, having determined that the recovery of the detected fault is not deferrable in view of the continued aggregation and analysis of the aggregated fault data, to change a state of the recovery of the detected fault from deferrable to undeferrable, and to schedule the recovery of the detected fault to be performed immediately.
15. A computer program product including a set of non-transitory, computer-readable media having instructions that, when executed by processing circuitry, cause the processing circuitry to perform a method of lifecycle handling of faults associated with a storage appliance, the method comprising:
recording and aggregating fault data pertaining to a plurality of detected faults associated with a storage appliance in a fault database;
analyzing the aggregated fault data to generate a recommendation for recovery of a detected fault from among the plurality of detected faults;
performing immediate or deferred recovery of the detected fault based at least on the recommendation, thereby reducing occurrences of the storage appliance being taken offline upon detection of a single fault;
determining that the detected fault is correctable; and
having determined that the detected fault is correctable, determining whether the recovery of the detected fault is deferrable or undeferrable,
wherein performing immediate recovery of the detected fault includes, having determined that the recovery of the detected fault is undeferrable, taking the storage appliance offline, and
wherein performing deferred recovery of the detected fault includes, having determined that the recovery of the detected fault is deferrable, scheduling the recovery of the detected fault to be performed at a later time.
16. The computer program product of claim 15 wherein the method further comprises:
having determined that the recovery of the detected fault is deferrable, continuing to aggregate fault data and analyze the aggregated fault data for subsequent detected faults associated with the storage appliance;
determining whether or not the recovery of the detected fault is deferrable in view of the continued aggregation and analysis of the aggregated fault data for the subsequent detected faults associated with the storage appliance;
having determined that the recovery of the detected fault is not deferrable in view of the continued aggregation and analysis of the aggregated fault data, changing a state of the recovery of the detected fault from deferrable to undeferrable; and
scheduling the recovery of the detected fault to be performed immediately.