Data protection in a mass storage system
View Patent ↗A method for distributing data over a set of N storage devices. The method includes apportioning the data into groups of (N−2) or fewer blocks, and adding to each group a parity block so as to form a parity set. The method further includes distributing the blocks of each parity set to the N storage devices so that each of the blocks is stored on a different storage device.
1. A method for distributing data over a set of N storage devices, wherein N is an integer greater than two, the method comprising:
apportioning the data into multiple groups, each group consisting of one to (N−2) blocks of data;
for each group, calculating a single parity block, so as to form a parity set of the blocks comprising the one to (N−2) blocks of data and the single parity block; and
distributing the blocks of each parity set over the N storage devices so that every storage device stores both parity blocks and data blocks, while for each parity set each of the blocks in the parity set is stored on a different storage device and at least one of the N storage devices stores none of the blocks in the parity set,
wherein distributing the blocks comprises distributing the single parity blocks among the N storage devices so that there is a balanced distribution of the parity blocks across the devices;
adding one or more storage devices to the N storage devices, thus forming an extended set of the storage devices; and
redistributing the blocks among the storage devices in the extended set so as to cause a portion of the blocks stored on the N devices to be redistributed to the one or more additional storage devices, while maintaining the balanced distribution and while ensuring that every storage device in the extended set stores both parity blocks and data blocks, while for each parity set each of the blocks in the parity set is stored on a different storage device and at least one of the storage devices in the extended set stores none of the blocks in the parity set, and without requiring a transfer of blocks between the storage devices in the N devices.
2. The method according to claim 1 , wherein redistributing the blocks comprises applying a consistent hashing function to determine the blocks that will be redistributed from the set of N storage devices to the extended set.
3. The method according to claim 1 , wherein redistributing the blocks comprises implementing a randomizing process to determine the blocks that will be redistributed from the set of N storage devices to the extended set.
4. A method for distributing data over a set of N storage devices, wherein N is an integer greater than two, the method comprising:
apportioning the data into multiple groups, each group consisting of one to (N−2) blocks of data;
for each group, calculating a single parity block, so as to form a parity set of the blocks comprising the one to (N−2) blocks of data and the single parity block; and
distributing the blocks of each parity set over the N storage devices so that every storage device stores both parity blocks and data blocks, while for each parity set each of the blocks in the parity set is stored on a different storage device and at least one of the N storage devices stores none of the blocks in the parity set,
wherein distributing the blocks comprises distributing the single parity blocks among the N storage devices so that there is a balanced distribution of the parity blocks across the devices;
removing a surplus device from the N storage devices, thus forming a depleted set of storage devices;
for each block on the surplus device, reconstructing the block by XOR-ing remaining blocks of the parity set of the block, so as to form a reconstructed block; and
redistributing the reconstructed blocks among the storage devices in the depleted set so as to cause the reconstructed blocks to be reallocated to the depleted set, while maintaining the balanced distribution and while ensuring that every storage device in the depleted set stores both parity blocks and data blocks, while for each parity set each of the blocks in the parity set is stored on a different storage device and at least one of the storage devices in the depleted set stores none of the blocks in the parity set, and without requiring a transfer of blocks among the storage devices in the depleted set.
5. The method according to claim 4 , wherein redistributing the reconstructed blocks comprises applying a consistent hashing function to distribute the reconstructed blocks to the depleted set.
6. The method according to claim 4 , wherein redistributing the reconstructed blocks comprises implementing a randomizing process to distribute the reconstructed blocks to the depleted set.