System state recovery in a distributed, cloud-based storage system
The system state recovery methods, systems and products disclosed herein enable an efficient means of recovering from a permanent site outage event in a distributed, block-based storage system. Embodiments teach using directory trees and journal updates for neighboring zones, which are still operational, as a means of recovering data for the site experiencing an outage. We further disclose load balancing techniques in order to improve efficiency of recovery. Load balancing is performed by selecting a leader zone and a group of non-leaders, which will comprise a set of recovery drivers. The systems within the set of recovery drivers are used to piece together lost data from the zone experiencing an outage. In embodiments, the systems, methods and products could be used with an Elastic Cloud System™.
1. A distributed block-based storage system having at least three zones, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising:
a. receiving an indication that a third zone has experienced an outage, the third zone having a plurality of directory trees;
b. choosing a set of recovery drivers to be used to recover the plurality of directory trees from the third zone, further comprising:
i. determining a tree update chronology for the first zone as a function of time;
ii. determining a tree update chronology for the second zone as a function of time;
iii. comparing the chronological tree update for the first zone with the chronological tree update for the second zone;
iv. selecting a leader zone based on the comparison, the leader zone having a chronological tree update extending further in time; and
v. determining if the non-leader zone can reach the leader zone by comparing a chronological journal update for the non-leader zone to the leader zone's chronological tree update, if the chronological journal update is at least as far in time as the leader zone's chronological tree update, adding the non-leader zone to the set of recovery drivers;
c. adding the leader zone to the set of recovery drivers; and
d. storing the set of recovery drivers in the memory.
2. The system of claim 1 wherein the processor is further configured to facilitate performance of operations further comprising recovering the third zone's plurality of directory trees using the set of recovery drivers.
3. The system of claim 1 wherein the processor is further configured to facilitate performance of operations further comprising distributing the workload across the set of recovery drivers by minimizing the number of times a zone in the set of recovery drivers is used to recover a directory tree in the third zone.
4. The system of claim 2 wherein the processor is further configured to facilitate performance of operations further comprising:
a. replaying the leader zone's tree journal;
b. collecting chronological data corresponding to journal updates for each of the zones in the set of recovery drivers;
c. chronologically ordering the chronological data for the journal updates for the zones in the set of recovery drivers; and
d. replaying a continuous stream of updates for zone three using the chronologically ordered journal updates for the zones in the set of recovery drivers.
5. A method of state recovery in a distributed block-based storage system having at least three zones comprising:
a. receiving an indication that a third zone has experienced an outage, the third zone having a plurality of directory trees;
b. choosing a set of recovery drivers to be used to recover the plurality of directory trees from the third zone, further comprising:
i. determining a tree update chronology for the first zone as a function of time;
ii. determining a tree update chronology for the second zone as a function of time;
iii. the chronological tree update for the first zone with the chronological tree update for the second zone;
iv. selecting a leader zone based on the comparison, the leader zone having a chronological tree update extending further in time; and
v. determining if the non-leader zone can reach the leader zone by comparing a chronological journal update for the non-leader zone to the leader zone's chronological tree update, if the chronological journal update is at least as far in time as the leader zone's chronological tree update, adding the non-leader zone to the set of recovery drivers;
c. adding the leader zone to the set of recovery drivers; and
d. storing the set of recovery drivers in the memory.
6. The method of claim 5 further comprising recovering the third zone's plurality of directory trees using the set of recovery drivers.
7. The method system of claim 5 further comprising distributing the workload across the set of recovery drivers by minimizing the number of times a zone in the set of recovery drivers is used to recover a directory tree in the third zone.
8. The method of claim 6 further comprising:
a. replaying the leader zone's tree journal;
b. collecting chronological data corresponding to journal updates for each of the zones in the set of recovery drivers;
c. chronologically ordering the chronological data for the journal updates for the zones in the set of recovery drivers; and
d. replaying a continuous stream of updates for zone three using the chronologically ordered journal updates for the zones in the set of recovery drivers.
9. A computer program product for state recovery in a distributed block-based storage system having at least three zones, the computer program product comprising: a computer readable storage medium having computer executable program code embodied therewith, the program code executable by a computer processor to:
a. receiving an indication that a third zone has experienced an outage, the third zone having a plurality of directory trees;
b. choosing a set of recovery drivers to be used to recover the plurality of directory trees from the third zone, further comprising:
i. determining a tree update chronology for the first zone as a function of time;
ii. determining a tree update chronology for the second zone as a function of time;
iii. comparing the chronological tree update for the first zone with the chronological tree update for the second zone;
iv. selecting a leader zone based on the comparison, the leader zone having a chronological tree update extending further in time; and
v. determining if the non-leader zone can reach the leader zone by comparing a chronological journal update for the non-leader zone to the leader zone's chronological tree update, if the chronological journal update is at least as far in time as the leader zone's chronological tree update, adding the non-leader zone to the set of recovery drivers;
c. adding the leader zone to the set of recovery drivers; and
d. storing the set of recovery drivers in the memory.
10. The computer program product of claim 9 wherein the program code executable by a computer processor is further configured to facilitate performance of operations further comprising recovering the third zone's plurality of directory trees using the set of recovery drivers.
11. The computer program product of claim 9 wherein the program code executable by a computer processor is further configured to facilitate performance of operations further comprising distributing the workload across the set of recovery drivers by minimizing the number of times a zone in the set of recovery drivers is used to recover a directory tree in the third zone.
12. The computer product of claim 10 wherein the program code executable by a computer processor is further configured to facilitate performance of operations further comprising:
a. replaying the leader zone's tree journal;
b. collecting chronological data corresponding to journal updates for each of the zones in the set of recovery drivers;
c. chronologically ordering the chronological data for the journal updates for the zones in the set of recovery drivers; and
d. replaying a continuous stream of updates for zone three using the chronologically ordered journal updates for the zones in the set of recovery drivers.