Method for lock-free clustered erasure coding and recovery of data across a plurality of data stores in a network
View Patent ↗The present invention provides a distributed clustering method to allow multiple active instances of consistency management processes that apply the same encoding scheme to be cooperative and function collectively. The techniques described herein facilitate an efficient method to apply an erasure encoding and decoding scheme across dispersed data stores that receive constant updates. The technique can be applied on many forms of distributed persistent data stores to provide failure resiliency and to maintain data consistency and correctness.
1. A method comprising:
providing:
a plurality of computer systems forming a distributed computer network, each computer system of the plurality of computer systems having a storage management process operating thereon;
wherein the plurality of storage management processes initialize into and operate as a cluster; and
a plurality of distributed persistent memories, each persistent memory of the plurality of distributed persistent memories interoperably coupled to at least one of the plurality of computer systems and operative to receive constant updates; and
each storage management process of the plurality of storage management processes managing data-set consistency and correctness;
wherein the step of managing data-set consistency and correctness comprises:
distributing, via an erasure encoding scheme, at least one data set across a plurality of the plurality of distributed persistent memories;
wherein each data set of the at least one data set comprises at least one data block and at least one checksum block;
wherein each data set of the at least one data set is managed by a single storage management process;
erasure decoding each data set of the at least one data set following an unsuccessful read operation; and
using a sequencing indicator to determine whether each data block and each checksum block of each data set of the at least one data set is modified completely.
2. The method of claim 1 , wherein each storage management processes comprises one data I/O processor and one erasure coding processor.
3. The method of claim 2 , wherein at least one erasure coding processor of the plurality of storage management processes shares the plurality of distributed, persistent memories for storing recovery blocks.
4. The method of claim 1 , comprising reinitializing the plurality of storage management processes responsive to at least one of:
adding a new storage management process to the plurality of storage management processes; and
removing a storage management process from the plurality of storage management processes.
5. The method of claim 1 , wherein each storage management process of the plurality of storage management processes communicates status information to other storage management process of the plurality of storage management processes.
6. The method of claim 1 , wherein the sequencing indicator is a sequence number entered into a header and trailer of each data block and checksum block of the data set.
7. The method of claim 1 , wherein the sequencing indicator is a sequence number entered into a register.
8. The method of claim 1 , wherein the sequencing indicator is a sequence number entered into a flag.
9. The method of claim 1 , wherein, during initialization of the cluster of the plurality of storage management processes, each storage management process:
ceases runtime processing and enters into an initialization mode;
determines data-set storage-management-process management responsibilities;
performs a recovery process; and
enters into an active mode.