High availability state machine and recovery
Embodiments of the present invention provide systems and methods for recovering a high availability storage system. The storage system includes a first layer and a second layer, each layer including a controller board, a router board, and storage elements. When a component of a layer fails, the storage system continues to function in the presence of a single failure of any component, up to two storage element failures in either layer, or a single power supply failure. While a component is down, the storage system will run in a degraded mode. The passive zone is not serving input/output requests, but is continuously updating its state in dynamic random access memory to enable failover within a short period of time using the layer that is fully operational. When the issue with the failed zone is corrected, a failback procedure brings the system back to a normal operating state.
1. A method of transitioning a degraded layer of a storage system to an active state, comprising:
attempting to boot a first layer and a second layer of the storage system to a quiescent state, wherein the first layer comprises a first local zone and a first remote zone, and the second layer comprises a second local zone and a second remote zone;
determining that the first layer booted successfully, and the second layer has failed due to an issue, wherein the second local zone and the second remote zone are in a down state;
performing a failover procedure from the second local zone to the first remote zone, wherein the first remote zone transitions to an active state;
responsive to the issue of the second layer being corrected, booting the second layer, wherein the second local zone and the second remote zone transition to the quiescent state;
transitioning the second local zone and the second remote zone to a passive state; and
performing a failback procedure from the first remote zone to the second local zone, wherein the first remote zone enters an active-quiescent state.
2. The method of claim 1 , further comprising operating the storage system in a degraded state after determining that the second layer has failed.
3. The method of claim 1 , further comprising transitioning the first remote zone to the passive state, wherein the storage system operates in a normal mode.
4. A method of performing failover on a layer of a storage system, comprising:
recovering an internal band of the storage system;
replaying metadata of the internal band until a last fully programmed stripe of the internal band;
identifying a first root of a first digest of the last fully programmed stripe of the internal band;
performing a group switch on the internal band;
recovering a system band of the storage system; and
flushing dirty map pages of a map table from dynamic random-access memory into the system band.
5. The method of claim 4 , further comprising:
replaying a hot band from a replay pointer to a last fully programmed stripe of the hot band;
attempting to locate a root of a digest of the last fully programmed stripe of a hot band flash block; and
responsive to the root being not located:
assigning a new flash block to the hot band;
rewriting valid flash pages to the new flash block, wherein the valid flash pages are between a beginning of a corrupt flash block and an end of the last fully programmed stripe of the hot band;
updating a map table to point to the valid flash pages;
validating data pages using a data page lookup on the map table; and
discarding the corrupt flash block.
6. The method of claim 4 , wherein identifying the first root of the first digest of the last fully programmed stripe of the internal band comprises:
responsive to the first root being not located:
replaying mirrored internal band write buffers from a mirrored write buffer pool; and
replaying mirrored internal band accumulation buffers responsive to internal band accumulation buffer content being not replayed.
7. The method of claim 6 , wherein replaying mirrored internal band write buffers from the mirrored write buffer pool comprises replaying mirrored internal band write buffers having an identification larger than an identification of a last successfully replayed flash page in flash blocks of the internal band.
8. The method of claim 4 , further comprising locating a second root of a second digest at an end of a last fully programmed stripe of a system flash block.
9. The method of claim 8 , further comprising:
responsive to the second root being not located:
assigning a new erased flash block to the system band;
rewriting valid map pages of the map table between a beginning of a corrupt system band metadata flash block and an end of a last fully programmed stripe of the system band, to a new system band metadata flash block; and
updating a map index of the map table to point to the valid map pages.
10. The method of claim 9 , further comprising:
using a bitmap to identify valid map pages;
cross-checking the valid map pages using a lookup on the map index; and
discarding the corrupt system band metadata flash block.
11. A storage system, comprising:
means for recovering an internal band of the storage system;
means for replaying metadata of the internal band until a last fully programmed stripe of the internal band;
means for identifying a first root of a first digest of the last fully programmed stripe of the internal band;
means for performing a group switch on the internal band;
means for recovering a system band of the storage system; and
means for flushing dirty map pages of a map table from dynamic random-access memory into the system band.
12. The storage system of claim 11 , further comprising:
means for replaying a hot band from a replay pointer to a last fully programmed stripe of the hot band;
means for attempting to locate a root of a digest of the last fully programmed stripe of a hot band flash block; and
responsive to the root being not located:
means for assigning a new flash block to the hot band;
means for rewriting valid flash pages to the new flash block, wherein the valid flash pages are between a beginning of a corrupt flash block and an end of the last fully programmed stripe of the hot band;
means for updating a map table to point to the valid flash pages;
means for validating data pages using a data page lookup on the map table; and
means for discarding the corrupt flash block.
13. The storage system of claim 11 , wherein the means for identifying the first root of the first digest of the last fully programmed stripe of the internal band comprises:
responsive to the first root being not located:
means for replaying mirrored internal band write buffers from a mirrored write buffer pool; and
means for replaying mirrored internal band accumulation buffers responsive to internal band accumulation buffer content being not replayed.
14. The storage system of claim 13 , wherein the means for replaying mirrored internal band write buffers from the mirrored write buffer pool comprises means for replaying mirrored internal band write buffers having an identification larger than an identification of a last successfully replayed flash page in flash blocks of the internal band.
15. The storage system of claim 11 , further comprising means for locating a second root of a second digest at an end of a last fully programmed stripe of a system flash block.
16. A storage system of claim 15 , further comprising:
responsive to the second root being not located:
means for assigning a new erased flash block to the system band;
means for rewriting valid map pages of the map table between a beginning of a corrupt system band metadata flash block and an end of a last fully programmed stripe of the system band, to a new system band metadata flash block; and
means for updating a map index of the map table to point to the valid map pages.
17. The storage system comprising means for performing the method of claim 1 .
18. The storage system of claim 17 , further comprising means for operating the storage system in a degraded state after determining that the second layer has failed.