Offloaded device-driven erasure coding
A method for storing data may include receiving user data at a group of storage devices, wherein the storage devices are interconnected, erasure coding the user data into redundancy blocks at the group of storage devices, and storing the redundancy blocks on at least two of the storage devices. The erasure encoding may be distributed among at least two of the storage devices. The redundancy blocks may be arranged in reliability groups. The redundancy blocks may be grouped by the storage devices independently of the partitioning of the user data by the user. The method may further include recovering data based on redundancy blocks. A storage device may include a storage medium, a network interface configured to communicate with one or more other storage devices, and a storage processing unit configured to erasure code user data into redundancy blocks cooperatively with the one or more other storage devices.
1. A method for storing data comprising:
receiving, using a network fabric, user data at two or more storage devices that are connected to the network fabric, wherein the two or more devices comprise two or more storage devices;
dividing, based on the receiving, using logic at one or more of the two or more devices, the user data into blocks comprising erasure coded data; and
storing at least a portion of the blocks on at least one of the two or more storage devices;
wherein the erasure coded data is based at least in part on a Reed-Solomon coding algorithm.
2. The method of claim 1 , wherein the two or more storage devices comprise one or more solid state storage devices, and wherein the two or more storage devices are connected to the network fabric through at least one network interface element, the network interface element being configured to support an NVMe-oF protocol.
3. The method of claim 2 , wherein:
the erasure coded data comprises at least one of a parity block or a parity chunk; and
the blocks are arranged by a storage management element.
4. The method of claim 3 , wherein at least one of the one or more solid state storage devices comprises a NAND flash storage device and a DRAM device.
5. The method of claim 4 , wherein at least one of the two or more storage devices is configured to perform one or more garbage collection operations.
6. The method of claim 5 , wherein the two or more storage devices are connected to one or more switches or hubs.
7. The method of claim 6 , wherein one or more storage devices of the two or more storage devices comprises an interconnect interface, the interconnect interface supporting a PCIe protocol.
8. The method of claim 7 , wherein at least one of the one or more switches or hubs is configured to interface via at least one of a PCIe protocol, a SATA protocol, or a SAS protocol.
9. The method of claim 8 , wherein at least one of the one or more solid state storage devices comprises one or more computational elements.
10. The method of claim 9 , wherein at least one of the one or more computational elements is a field programmable gate array (FPGA) or an application specific integrated circuit ASIC device.
11. An apparatus comprising:
a data storage system comprising:
a fabric;
two or more devices connected to the fabric, wherein the two or more devices comprise a first storage device and a second storage device connected to the fabric;
an erasure coding logic at one or more of the two or more devices configured to erasure-code and divide user data into a first erasure coded data and a second erasure coded data based on receiving the user data at one or more of the two or more devices; and
a network interface configured to couple the first storage device and the second storage device to the fabric, wherein the fabric supports an NVMe-oF protocol;
wherein the first erasure coded data is based at least in part on a Reed-Solomon coding algorithm.
12. The data storage system of claim 11 , wherein the first storage device comprises a solid state storage device.
13. The data storage system of claim 12 , wherein the first erasure coded data comprises at least one of a parity block or a parity chunk.
14. The data storage system of claim 13 , wherein:
the solid state storage device comprises a NAND flash storage device and a DRAM device; and
the first storage device is configured to perform a garbage collection operation.
15. The data storage system of claim 14 , wherein the first storage device is connected to one or more switches or hubs.
16. The data storage system of claim 15 , wherein:
the first storage device is configured to support a PCIe protocol; and
at least one of the one or more switches or hubs is configured to interface via at least one of a PCIe protocol, a SATA protocol, or a SAS protocol.
17. The data storage system of claim 16 , wherein the solid state storage device comprises one or more computational elements.
18. The data storage system of claim 17 , wherein at least one of the one or more computational elements is a field programmable gate array (FPGA) or an application specific integrated circuit ASIC device.