Systems, methods, and devices for storage shuffle acceleration
A method of processing data in a system having a host and a storage node may include performing a shuffle operation on data stored at the storage node, wherein the shuffle operation may include performing a shuffle write operation, and performing a shuffle read operation, wherein at least a portion of the shuffle operation is performed by an accelerator at the storage node. A method for partitioning data may include sampling, at a device, data from one or more partitions based on a number of samples, transferring the sampled data from the device to a host, determining, at the host, one or more splitters based on the sampled data, communicating the one or more splitters from the host to the device, and partitioning, at the device, data for the one or more partitions based on the one or more splitters.
1. A method of processing data, the method comprising:
performing a shuffle operation, wherein the shuffle operation comprises:
performing, at a storage node, at least a portion of a shuffle write operation, wherein the at least a portion of the shuffle write operation comprises storing output data from a map operation at the storage node; and
performing, at the storage node, at least a portion of a shuffle read operation, wherein the at least a portion of the shuffle read operation comprises reading at least a portion of the output data from the map operation at the storage node;
wherein at least a portion of the shuffle operation is performed by an accelerator at the storage node.
2. The method of claim 1 , wherein the at least a portion of the shuffle write operation comprises a partition operation.
3. The method of claim 1 , wherein the at least a portion of the shuffle write operation comprises one or more of an aggregation operation, a sort operation, a merge operation, a serialize operation, a compression operation, or a spill operation.
4. The method of claim 1 , wherein the at least a portion of the shuffle read operation comprises one or more of a fetching operation, a decompression operation, a deserialize operation, a merge operation, a sort operation, or an aggregation operation.
5. The method of claim 1 , wherein the at least a portion of the shuffle operation performed at the storage node comprises a partition operation performed using a peer-to-peer (P2P) connection between an accelerator and a storage device at the storage node.
6. The method of claim 1 , wherein the portion of the shuffle operation performed at the storage node comprises a data spill operation performed using a P2P connection between the accelerator and a storage device at the storage node.
7. The method of claim 1 , wherein the at least a portion of the shuffle operation performed by the accelerator at the storage node comprises a fetch operation performed using a direct memory access operation.
8. The method of claim 1 , wherein the at least a portion of the shuffle operation performed by the accelerator at the storage node comprises a data merge operation performed using a P2P connection between the accelerator and a storage device at the storage node.
9. A storage node comprising:
a storage device; and
an accelerator;
wherein the storage node is configured to perform at least a portion of a shuffle operation, the at least a portion of the shuffle operation comprising:
performing, at the storage node, at least a portion of a shuffle write operation, wherein the at least a portion of the shuffle write operation comprises writing output data from a map operation to the storage device; and
performing, at the storage node, at least a portion of a shuffle read operation, wherein the at least a portion of the shuffle read operation comprises reading, from the storage device, at least a portion of the output data from the map operation; and
wherein the storage node is configured to perform at least a portion of the at least a portion of the shuffle operation using the accelerator.
10. The storage node of claim 9 , wherein:
the storage node further comprises a peer-to-peer (P2P) connection between the storage device and the accelerator; and
the storage device and the accelerator are configured to perform the at least a portion of the at least a portion of the shuffle operation by transferring data over the P2P connection.
11. The storage node of claim 10 , wherein:
the accelerator is integral with the storage device; and
the storage node comprises a server.
12. The storage node of claim 10 , wherein the storage device is a first storage device, the accelerator is a first accelerator, the P2P connection is a first P2P connection, and the storage node further comprises:
a second storage device;
a second accelerator; and
a second peer-to-peer (P2P) connection between the second storage device and the second accelerator;
wherein the second storage device and the second accelerator are configured to perform the at least a portion of the at least a portion of the shuffle operation by transferring data over the second P2P connection.
13. The storage node of claim 12 , wherein the first and second storage devices are configured to perform the at least a portion of the at least a portion of the shuffle operation by transferring data through a direct memory access operation.
14. The storage node of claim 9 , wherein the storage node is configured to perform the at least a portion of the at least a portion of the shuffle operation by transferring data to an additional storage node through a remote direct memory access operation.
15. A method for partitioning data, the method comprising:
sampling, at a device, data from one or more partitions to generate sampled data;
transferring the sampled data from the device to a host;
determining, at the host, one or more splitters based on the sampled data;
communicating the one or more splitters from the host to the device; and
partitioning, at the device, data for the one or more partitions based on the one or more splitters;
wherein the sampling comprises reading a portion of the data from the one or more partitions.
16. The method of claim 15 , wherein the sampling is based on a number of samples, the method further comprising:
determining, at the device, a number of records for the one or more partitions;
communicating the number of records for the one or more partitions from the device to the host;
determining, at the host, the number of samples for the one or more partitions; and
communicating the number of samples from the host to the device.
17. The method of claim 15 , wherein the sampling is performed by an accelerator at the device.
18. A system comprising:
a storage node comprising an accelerator; and
a host configured to perform a first portion of a shuffle operation
wherein the storage node is configured to perform a second portion of the shuffle operation, the second portion of the shuffle operation comprising:
performing, at the storage node, at least a portion of a shuffle write operation, wherein the at least a portion of the shuffle write operation comprises writing output data from a map operation at the storage node; and
performing, at the storage node, at least a portion of a shuffle read operation, wherein the at least a portion of the shuffle read operation comprises reading at least a portion of the output data from the map operation at the storage node; and
wherein the storage node is configured to perform at least a portion of the second portion of the shuffle operation using the accelerator.