Parity generation offload using peer-to-peer data transfers in data storage system
A system and method improve the performance of non-volatile memory storage by offloading parity computations to facilitate high speed data transfers, including direct memory access (DMA) transfers, between a remote host and a non-volatile memory based storage system, such as a flash memory based data storage device (e.g., SSD). In conjunction with writing to non-volatile memory storage, a stripe map is used to target a selected data storage device for parity generation. All data of a stripe is transmitted to the selected data storage device to generate the parity and the generated parity is propagated from the selected data storage device to other data storage devices in the stripe. The data for the stripe may also be propagated from the selected data storage device to the other data storage devices in the stripe.
1. A method of managing data storage in a non-volatile memory system comprising a plurality of data storage devices, the method comprising:
receiving, at a controller system, a compaction request to compact valid blocks of a first Redundancy Coding stripe and valid blocks of a second Redundancy Coding stripe into a single third Redundancy Coding stripe,
the first Redundancy Coding stripe, the second Redundancy Coding stripe and the third Redundancy Coding stripe including storage locations in a set of data storage devices comprising three or more of the plurality of data storage devices;
in response to receiving the compaction request, the controller system performing a sequence of operations, including:
identifying a first data storage device and a second data storage device in the set of data storage devices from which to read data for the first Redundancy Coding stripe and the second Redundancy Coding stripe, the identified first data storage device and the second data storage device each including a controller, non-volatile memory and a data buffer;
identifying a third data storage device in the set of data storage devices at which to compute parity for the identified third Redundancy Coding stripe, the identified third data storage device including a controller, non-volatile memory and a data buffer;
sending one or more data transfer commands to the first data storage device and the second data storage device to locally copy valid blocks from the first Redundancy Coding stripe and the second Redundancy Coding stripe to the third Redundancy Coding stripe within the respective data storage devices, whereby the third Redundancy Coding stripe includes storage locations in each of the first and second data storage devices; and
sending a parity rebuild command to the third data storage device to rebuild parity for the third Redundancy Coding stripe,
wherein rebuilding parity for the third Redundancy Coding stripe includes obtaining, at the third data storage device, data in the third Redundancy Coding stripe from the first data storage device using peer-to-peer data transfers from the first data storage device to the third data storage device and obtaining, at the third data storage device, data in the third Redundancy Coding stripe from the second data storage device using peer-to-peer data transfers from the second data storage device to the third data storage device.
2. The method of claim 1 , wherein sending the one or more data transfer commands includes:
sending a first data transfer command to the first data storage device to locally copy valid blocks from the first Redundancy Coding stripe to the third Redundancy Coding stripe within the first data storage device;
sending a second data transfer command to the second data storage device to locally copy valid blocks from the first Redundancy Coding stripe to the third Redundancy Coding stripe within the second data storage device;
sending a third data transfer command to the first data storage device to locally copy valid blocks from the second Redundancy Coding stripe to the third Redundancy Coding stripe within the first data storage device; and
sending a fourth data transfer command to the second data storage device to locally copy valid blocks from the second Redundancy Coding stripe to the third Redundancy Coding stripe, within the second data storage device.
3. The method of claim 2 , wherein the first data transfer command, the second data transfer command, the third data transfer command and the fourth data transfer command are configured to cause a copy of the valid blocks to be sent to one or more volatile memory buffers on the third data storage device.
4. The method of claim 3 , wherein the parity rebuild command instructs the third data storage device to rebuild parity for the third Redundancy Coding stripe from the data in the one or more volatile memory buffers.
5. The method of claim 4 , wherein the parity rebuild command further instructs the third data storage device to locally write the computed parity, within the third data storage device, to the third Redundancy Coding stripe.
6. The method of claim 1 , wherein the parity rebuild command further instructs the third data storage device to compute parity for the third Redundancy Coding stripe.
7. The method of claim 6 , wherein the parity rebuild command further instructs the third data storage device to locally write the computed parity, within the third data storage device, to the third Redundancy Coding stripe.
8. The method of claim 1 , wherein:
the controller system is a host system external to the non-volatile memory system; and
the compaction request is received from an application executed by the host system.
9. The method of claim 1 , wherein the controller system receives the compaction request from a host system external to the non-volatile memory system.
10. The method of claim 1 , wherein the controller system is one or more modules of a respective data storage device of the plurality of data storage devices.
11. The method of claim 1 , wherein:
the controller system receives a plurality of compaction requests, and
different ones of the data storage devices compute parity for respective ones of the plurality of compaction requests.
12. A memory controller, comprising:
a storage interface means for coupling the memory controller to a plurality of data storage devices;
a communication interface means for receiving compaction requests, each compaction request comprising a request to compact valid blocks of a first Redundancy Coding stripe and valid blocks of a second Redundancy Coding stripe into a single third Redundancy Coding stripe, the first Redundancy Coding stripe, the second Redundancy Coding stripe and the third Redundancy Coding stripe including storage locations in a set of data storage devices comprising three or more of the plurality of data storage devices; and
a command processing means for processing a respective compaction request by performing a sequence of operations, including:
means for identifying a first data storage device and a second data storage device in the set of data storage devices from which to read data for the first Redundancy Coding stripe and the second Redundancy Coding stripe, the identified first data storage device and the second data storage device including a controller, non-volatile memory and a data buffer;
means for identifying a third data storage device in the set of data storage devices at which to compute and store parity for the identified third Redundancy Coding stripe, the identified third data storage device including a controller, non-volatile memory and a data buffer;
means for sending one or more data transfer commands to the first data storage device and the second data storage device to locally copy valid blocks from the first Redundancy Coding stripe and the second Redundancy Coding stripe to the third Redundancy Coding stripe within the respective data storage devices, whereby the third Redundancy Coding stripe includes storage locations in each of the first and second data storage devices; and
means for sending a parity rebuild command to the third data storage device to rebuild parity for the third Redundancy Coding stripe,
wherein rebuilding parity for the third Redundancy Coding stripe includes obtaining, at the third data storage device, data in the third Redundancy Coding stripe from the first data storage device using peer-to-peer data transfers from the first data storage device to the third data storage device and obtaining, at the third data storage device, data in the third Redundancy Coding stripe from the second data storage device using peer-to-peer data transfers from the second data storage device to the third data storage device.
13. The memory controller of claim 12 , wherein each data storage device of the plurality of data storage devices includes:
non-volatile memory for durably storing information, one or more data buffers for temporarily storing information being written to or read from the non-volatile memory of the data storage device; and
a controller for controlling operation of the data storage device, including execution of compaction commands.
14. The memory controller of claim 12 , wherein:
the non-volatile memory of each data storage device of the plurality of data storage devices comprises flash memory;
and the one or more data buffers of each data storage device of the plurality of data storage devices comprises non-volatile RAM.
15. The memory controller of claim 12 , wherein sending the one or more data transfer commands includes:
sending a first data transfer command to the first data storage device to locally copy valid blocks from the first Redundancy Coding stripe to the third Redundancy Coding stripe within the first data storage device;
sending a second data transfer command to the second data storage device to locally copy valid blocks from the first Redundancy Coding stripe to the third Redundancy Coding stripe within the second data storage device;
sending a third data transfer command to the first data storage device to locally copy valid blocks from the second Redundancy Coding stripe to the third Redundancy Coding stripe within the first data storage device; and
sending a fourth data transfer command to the second data storage device to locally copy valid blocks from the second Redundancy Coding stripe to the third Redundancy Coding stripe, within the second data storage device.
16. The memory controller of claim 15 , wherein the first data transfer command, the second data transfer command, the third data transfer command and the fourth data transfer command are configured to cause a copy of the valid blocks to be sent to one or more volatile memory buffers on the third data storage device.
17. The memory controller of claim 16 , wherein the parity rebuild command instructs the third data storage device to rebuild parity for the third Redundancy Coding stripe from the data in the one or more volatile memory buffers.
18. The memory controller of claim 17 , wherein the parity rebuild command further instructs the third data storage device to locally write the computed parity, within the third data storage device, to the third Redundancy Coding stripe.
19. The memory controller of claim 12 , wherein:
the memory controller is a host system external to the non-volatile memory system; and
the compaction request is received from an application executed by the host system.
20. The memory controller of claim 12 , wherein the memory controller receives the compaction request from a host system external to the non-volatile memory system.
21. A computer readable storage medium storing one or more programs configured for execution by a memory controller configured to be coupled to a plurality of data storage devices, the one or more programs comprising instructions that when executed by one or more processors of the memory controller, cause the memory controller to:
receive compaction requests, each compaction request comprising a request to compact valid blocks of a first Redundancy Coding stripe and valid blocks of a second Redundancy Coding stripe into a single third Redundancy Coding stripe, the first Redundancy Coding stripe, the second Redundancy Coding stripe and the third Redundancy Coding stripe including storage locations in a set of data storage devices comprising three or more of the plurality of data storage devices; and
process a respective compaction request by performing a sequence of operations, including:
identifying a first data storage device and a second data storage device in the set of data storage devices from which to read data for the first Redundancy Coding stripe and the second Redundancy Coding stripe, the identified first data storage device and the second data storage device including a controller, non-volatile memory and a data buffer;
identifying a third data storage device in the set of data storage devices at which to compute and store parity for the identified third Redundancy Coding stripe, the identified third data storage device including a controller, non-volatile memory and a data buffer;
sending one or more data transfer commands to the first data storage device and the second data storage device to locally copy valid blocks from the first Redundancy Coding stripe and the second Redundancy Coding stripe to the third Redundancy Coding stripe within the respective data storage devices, whereby the third Redundancy Coding stripe includes storage locations in each of the first and second data storage devices; and
sending a parity rebuild command to the third data storage device to rebuild parity for the third Redundancy Coding stripe,
wherein rebuilding parity for the third Redundancy Coding stripe includes obtaining, at the third data storage device, data in the third Redundancy Coding stripe from the first data storage device using peer-to-peer data transfers from the first data storage device to the third data storage device and obtaining, at the third data storage device, data in the third Redundancy Coding stripe from the second data storage device using peer-to-peer data transfers from the second data storage device to the third data storage device.