Offloaded device-driven erasure coding
View Patent ↗A method for storing data may include receiving user data at a group of storage devices, wherein the storage devices are interconnected, erasure coding the user data into redundancy blocks at the group of storage devices, and storing the redundancy blocks on at least two of the storage devices. The erasure encoding may be distributed among at least two of the storage devices. The redundancy blocks may be arranged in reliability groups. The redundancy blocks may be grouped by the storage devices independently of the partitioning of the user data by the user. The method may further include recovering data based on redundancy blocks. A storage device may include a storage medium, a network interface configured to communicate with one or more other storage devices, and a storage processing unit configured to erasure code user data into redundancy blocks cooperatively with the one or more other storage devices.
1. A method for storing data comprising:
receiving user data at two or more storage devices that are connected by a network fabric;
dividing the user data into blocks comprising erasure coded data at the two or more storage devices; and
storing the blocks on at least two of the storage devices;
wherein the blocks are arranged by the storage devices.
2. The method of claim 1 , wherein:
the blocks are arranged in reliability groups; and
one or more of the storage devices maintains metadata about the reliability groups of its blocks.
3. The method of claim 1 , wherein:
at least one of the storage devices is capable of computational storage; and
a user partitions the user data between the storage devices based on computations to be performed on the user data.
4. The method of claim 1 , wherein at least one of the storage devices sends a reliable write acknowledgment based on the erasure coded data.
5. The method of claim 1 , further comprising recovering data based on live blocks.
6. The method of claim 1 , wherein one or more of the blocks are self-descriptive to facilitate recovering data.
7. The method of claim 1 , wherein:
the blocks are arranged in reliability groups;
one or more of the storage devices informs at least one other storage device having a block in a reliability group about a block in the reliability group that is to be erased; and
one of the storages devices erases the block based on receiving a proceed signal from another storage device having the block in the reliability group.
8. The method of claim 1 , wherein a first storage device of the two or more storage devices informs a second storage device of the two or more storage devices about a block that is to be erased.
9. The method of claim 8 , wherein the first storage device erases the block based on receiving a proceed signal from the second storage device.
10. The method of claim 1 , wherein:
the blocks are arranged in reliability groups;
a first storage device of the two or more storage devices:
is informed by a second storage device of the two or more storage devices that a block in a reliability group is to be erased by the second storage device; and
examines its own redundancy block in the reliability group.
11. The method of claim 10 , wherein the first storage device sends a proceed signal to the second storage device.
12. The method of claim 10 , wherein the first storage device sends a request to one or more other storage devices having a block in the reliability group that its own block in the reliability group be added to a new reliability group.
13. The method of claim 1 , wherein:
the blocks are arranged in reliability groups; and
the two or more storage devices recalculates parity for a reliability group based on erasure of a block in the reliability group.
14. A storage device comprising:
a storage medium;
a network interface configured to communicate with one or more other storage devices through a network fabric; and
a storage processing unit configured to divide user data into blocks comprising erasure coded data cooperatively with the one or more other storage devices.
15. The storage device of claim 14 , wherein the storage processing unit is configured to recover data based on one or more of the blocks.
16. The storage device of claim 14 , wherein the storage processing unit is configured to initiate a parity recalculation based on a garbage collection operation on one or more of the blocks.
17. The storage device of claim 14 , wherein:
the blocks are arranged in reliability groups; and
the storage processing unit is configured to:
inform one or more other storage devices having a block in a reliability group about a block in the reliability group that is to be erased; and
erase the block based on receiving a proceed signal from the one or more other storage devices having the block in the reliability group.
18. A data storage system comprising:
a first storage device that is fabric-enabled; and
a second storage device that is fabric-enabled and connected through network fabric to the first storage device;
wherein the first and second storage devices are configured to:
receive user data from a user; and
cooperatively divide the user data into blocks comprising erasure coded data.
19. The data storage system of claim 18 , wherein:
the user data is partitioned between the storage devices by a user; and
the first and second storage devices are configured to arrange the user data into blocks independently of the partitioning of the user data by the user.
20. The data storage system of claim 18 , wherein the first and second storage devices are configured to recover data based on one or more of the blocks.