FPGA acceleration system for MSR codes

1. An apparatus comprising:

a host interface circuit configured to receive an offloading instruction from a host processing device configured to manage a distributed storage system, wherein the offloading instruction instructs the apparatus to compute an error correction code associated with data elements stored by an external distributed storage system;

a memory interface circuit configured to receive the data elements from the external distributed storage system; and

an error code computation circuit configured to, at least in part, compute the error correction code without additional processing by the host processing device.

2. The apparatus of claim 1 , wherein the apparatus is configured to compute an error correction code by:

partitioning the data elements into sub-portions;

executing multiple passes, by sub-portion, of the data elements; and

aggregating an output of each pass to form the error correction code.

3. The apparatus of claim 1 , wherein the memory interface circuit is configured to automatically write an output of the error code computation circuit to an external memory.

4. The apparatus of claim 1 , wherein the data elements are associated with a number of data nodes of the distributed storage system, wherein a data node stores a portion of the data elements at a remote location; and

wherein the error code computation circuit is reconfigurable based upon the number of data nodes.

5. The apparatus of claim 1 , wherein the error code computation circuit comprises:

a tree of alternating multiplication circuits and additional circuits.

6. The apparatus of claim 5 , wherein the multiplication circuits comprise lookup tables; and

wherein the additional circuits comprise XOR gates.

7. The apparatus of claim 1 , wherein the error code computation circuit is reprogrammable by the host processing device.

8. The apparatus of claim 1 , wherein the error code comprises a Minimum Storage Regenerating (MSR) code.

9. A system comprising:

a memory configured to store data elements stored by an external distributed storage system;

a host processing device configured to configured to manage a distributed storage system and offload a computation of an error correction code to a reprogrammable processing device; and

the reprogrammable processing device configured to compute the error correction code, wherein the reprogrammable processing device comprises:

a reconfigurable pipeline that computes, at least in part, the error correction code and is reconfigured based upon a number of data elements.

10. The system of claim 9 , wherein the host processing device is configured to, during operation:

execute a computation thread of instructions that generates a request to offload the computation of an error correction code to the reprogrammable processing device, and

create a callback thread of instructions to control the reprogrammable processing device, and

create at least one input/output (I/O) thread of instructions for either reading the data elements from or reading the error correction code to the memory.

11. The system of claim 10 , wherein, when the host processing device is configured to, when generating a parity data element:

halt the computation thread in response to enqueuing an offload request;

in response to the callback thread indicating that the reprogrammable processing device has computed the error correction code, restarting the computation thread; and

cause the I/O thread to write the error correction code to the memory as the parity data element.

12. The system of claim 11 , wherein, when the host processing device is configured to, when reconstructing a data element:

cause the I/O thread to read data elements from the memory to the reprogrammable processing device;

halt the computation thread until the I/O thread indicates that the reprogrammable processing device has received the data elements; and

notifying a user thread of instructions that the data element has been reconstructed in response to the reprogrammable processing device has computed the error correction code.

13. The system of claim 9 , wherein the reprogrammable processing device comprises:

a host interface circuit configured to receive an offloading instruction from the host processing device;

a memory interface circuit configured to receive the data elements;

a memory buffer circuit configured to temporarily store the data elements;

an error code computation circuit configured to, at least in part, compute the error correction code without additional processing by the host processing device.

14. The system of claim 9 , wherein the host processing device and the reprogrammable processing device are configured to communicate via direct memory access.

15. The system of claim 9 , wherein the memory and the reprogrammable processing device are configured to communicate via direct memory access.

16. An apparatus comprising:

a host processing device configured to offload a computation of an error correction code to an offload processing device;

an input/output (I/O) system configured to read and write data elements to a system of distributed storage devices; and

the offload processing device configured to compute the error correction code based, at least in part, upon data elements stored by the distributed storage devices, wherein the offload processing device comprises:

a memory interface circuit configured to receive the data elements; and

an error code computation circuit configured to, at least in part, compute the error correction code without additional processing by the host processing device.

17. The apparatus of claim 16 , wherein the offload processing device is configured to compute an error correction code by:

partitioning the data elements into sub-portions;

executing multiple passes, by sub-portion, of the data elements; and

aggregating an output of each pass to form the error correction code.

18. The apparatus of claim 16 , wherein the data elements are associated with a number of storage devices; and

wherein a number within the error code computation circuit is reconfigurable to match the number of data nodes, such that, there is a correlation between the data nodes and the error code computation circuit.

19. The apparatus of claim 16 , wherein the error code computation circuit comprise a tree of alternating multiplication circuits and addition circuits; and

wherein the multiplication circuits comprise lookup tables, and

wherein the additional circuits comprise XOR gates.

20. The apparatus of claim 16 , wherein the offload processing device is configured to:

in response to an encode request, compute the error correction code as a parity element, and

in response to a decode request, compute the error correction code as a rebuilt data element.