Mirrored block with quorum set management for use in tracking valid mirrors of a journal
Implementations are provided herein for using a Node State Block (“NSB”) to track the validity of various copies of the journal. The NSB can contain states local to the node and be implemented independent of the cluster state, and be readable at node boot time. The NSB can be mirrored to every storage drive within the node. The NSB can contain a generation number and a validity bit. Every time the NSB is updated, the generation number can be incremented. An update must be written to a quorum of drives to be considered committed. At mount time, the NSB is read from all the drives, and the copy with the highest generation number is used. If a quorum of drives cannot be read, the mount fails. Read quorum, or the quorum required at mount when reading the NSB across the set of storage drives requires a strict majority of the storage drives to have valid copies of the NSB. Write quorum, or the quorum required to update the NSB across all storage drives, requires a simple majority of drives.
1. A method comprising:
maintaining a node state block (“NSB”) within a node among a cluster of nodes, wherein the NSB contains a journal generation set and a set of Globally Unique identifiers (“GUIDs”) corresponding to storage drives of the node;
maintaining a drive local copy of the NSB on each storage drive of the node; and
determining a valid journal state based on the set of the drive local copies of the NSB, wherein determining the valid journal state includes determining a read quorum status and write quorum status of all the node's storage drives, wherein a read quorum status corresponds to a majority of the node's storage drives having valid copies of the NSB and the write quorum status requires at least a majority of the node's storage drives to be available.
2. The method of claim 1 , further comprising:
determining a first read quorum state based on the set of the drive local copies of the NSB;
in response to the first read quorum state achieving quorum, reading the journal generation set from each drive local copy of the NSB;
in response to determining that the journal generation set on a storage drive of the node has at least one of a read error, a checksum failure, or inconsistent internal generation numbers, excluding the drive from a second read quorum state; and
in response to the second read quorum state passing quorum, setting the NSB to the journal generation set to a most recent NSB version associated with the second read quorum state.
3. The method of claim 2 , further comprising:
overwriting the journal generation set of the drive local copy of the NSB to the most recent NSB version for each storage drive of the node whose drive local copy of the NSB had a state of failed reads, bad data, or torn writes;
in response to determining a third read quorum state passes quorum, updating the drive local copy of the NSB to the most recent NSB version for each storage drive of the node whose drive local copy of the NSB had inconsistent internal generation numbers;
in response to determining a fourth read quorum state passes quorum, processing two NSB updates;
in response to determining a fifth read quorum state passes quorum, completing any updates to the set of GUIDs in the NSB which were interrupted by a panic;
in response to determining a sixth read quorum state passes quorum, mounting the node.
4. The method of claim 3 , further comprising:
in response to determining one of the first read quorum state, the second read quorum state, the third read quorum state, the fourth read quorum state, the fifth read quorum state, or the sixth read quorum failing to pass quorum, failing the mount.
5. The method of claim 1 , further comprising:
taking an exclusive lock on the NSB;
incrementing the journal generation set; and
in response to determining a first write quorum state passes quorum, overwriting a drive local copy of the NSB on each storage drive of the node with the incremented journal generation set.
6. The method of claim 1 , further comprising:
identifying a new storage drive of the node; and
in response to determining the drive local copy of the NSB of the new storage drive is out of date, overwriting the drive local copy of the NSB of the new storage drive with the NSB.
7. A system comprising at least one storage device and at least one hardware processor configured to:
maintain a node state block (“NSB”) within a node among a cluster of nodes, wherein the NSB contains a journal generation set and a set of Globally Unique identifiers (“GUIDs”) corresponding to storage drives of the node;
maintain a drive local copy of the NSB on each storage drive of the node; and
determine a valid journal state based on the set of the drive local copies of the NSB, wherein determining the valid journal state includes determining a read quorum status and write quorum status of all the node's storage drives, wherein a read quorum status corresponds to a majority of the node's storage drives having valid copies of the NSB and the write quorum status requires at least a majority of the node's storage drives to be available.
8. The system of claim 7 , further configured to:
determine a first read quorum state based on the set of the drive local copies of the NSB;
in response to the first read quorum state achieving quorum, read the journal generation set from each drive local copy of the NSB;
in response to determining that the journal generation set on a storage drive of the node has at least one of a read error, a checksum failure, or inconsistent internal generation numbers, exclude the drive from a second read quorum state; and
in response to the second read quorum state passing quorum, set the NSB to the journal generation set to a most recent NSB version associated with the second read quorum state.
9. The system of claim 8 , further configured to:
overwrite the journal generation set of the drive local copy of the NSB to the most recent NSB version for each storage drive of the node whose drive local copy of the NSB had a state of failed reads, bad data, or torn writes;
in response to determining a third read quorum state passes quorum, update the drive local copy of the NSB to the most recent NSB version for each storage drive of the node whose drive local copy of the NSB had inconsistent internal generation numbers;
in response to determining a fourth read quorum state passes quorum, process two NSB updates;
in response to determining a fifth read quorum state passes quorum, complete any updates to the set of GUIDs in the NSB which were interrupted by a panic;
in response to determining a sixth read quorum state passes quorum, mount the node.
10. The system of claim 9 , further configured to:
in response to determining one of the first read quorum state, the second read quorum state, the third read quorum state, the fourth read quorum state, the fifth read quorum state, or the sixth read quorum failing to pass quorum, fail the mount.
11. The system of claim 7 , further configured to:
take an exclusive lock on the NSB;
increment the journal generation set; and
in response to determining a first write quorum state passes quorum, overwrite a drive local copy of the NSB on each storage drive of the node with the incremented journal generation set.
12. The system of claim 7 , further configured to:
identify a new storage drive of the node; and
in response to determining the drive local copy of the NSB of the new storage drive is out of date, overwrite the drive local copy of the NSB of the new storage drive with the NSB.
13. A non-transitory computer readable medium with program instructions stored thereon to perform the following acts:
maintaining a node state block (“NSB”) within a node among a cluster of nodes, wherein the NSB contains a journal generation set and a set of Globally Unique identifiers (“GUIDs”) corresponding to storage drives of the node;
maintaining a drive local copy of the NSB on each storage drive of the node; and
determining a valid journal state based on the set of the drive local copies of the NSB, wherein determining the valid journal state includes determining a read quorum status and write quorum status of all the node's storage drives, wherein a read quorum status corresponds to a majority of the node's storage drives having valid copies of the NSB and the write quorum status requires at least a majority of the node's storage drives to be available.
14. The non-transitory computer readable medium of claim 13 , with program instructions stored thereon to further perform the following acts:
determining a first read quorum state based on the set of the drive local copies of the NSB;
in response to the first read quorum state achieving quorum, reading the journal generation set from each drive local copy of the NSB;
in response to determining that the journal generation set on a storage drive of the node has at least one of a read error, a checksum failure, or inconsistent internal generation numbers, excluding the drive from a second read quorum state; and
in response to the second read quorum state passing quorum, setting the NSB to the journal generation set to a most recent NSB version associated with the second read quorum state.
15. The non-transitory computer readable medium of claim 14 , with program instructions stored thereon to further perform the following acts:
overwriting the journal generation set of the drive local copy of the NSB to the most recent NSB version for each storage drive of the node whose drive local copy of the NSB had a state of failed reads, bad data, or torn writes;
in response to determining a third read quorum state passes quorum, updating the drive local copy of the NSB to the most recent NSB version for each storage drive of the node whose drive local copy of the NSB had inconsistent internal generation numbers;
in response to determining a fourth read quorum state passes quorum, processing two NSB updates;
in response to determining a fifth read quorum state passes quorum, completing any updates to the set of GUIDs in the NSB which were interrupted by a panic;
in response to determining a sixth read quorum state passes quorum, mounting the node.
16. The non-transitory computer readable medium of claim 15 , with program instructions stored thereon to further perform the following acts:
in response to determining one of the first read quorum state, the second read quorum state, the third read quorum state, the fourth read quorum state, the fifth read quorum state, or the sixth read quorum failing to pass quorum, failing the mount.
17. The non-transitory computer readable medium of claim 13 , with program instructions stored thereon to further perform the following acts:
taking an exclusive lock on the NSB;
incrementing the journal generation set; and
in response to determining a first write quorum state passes quorum, overwriting a drive local copy of the NSB on each storage drive of the node with the incremented journal generation set.
18. The non-transitory computer readable medium of claim 13 , with program instructions stored thereon to further perform the following acts:
identifying a new storage drive of the node; and
in response to determining the drive local copy of the NSB of the new storage drive is out of date, overwriting the drive local copy of the NSB of the new storage drive with the NSB.