System and method for offloading computation to storage nodes in distributed system

1. A distributed computing system, the system comprising:

a compute cluster comprising one or more compute nodes;

a storage cluster comprising a plurality of storage nodes; and

a master node for distributing data among the storage nodes;

wherein a respective storage node comprises:

a processor;

a memory;

a receiving module configured to receive, from a compute node, computation instructions associated with a computation task, wherein the computation task is divided into a number of sub-tasks in such a way that each sub-task only requires data stored on a single storage node;

a computation module configured to execute a corresponding sub-task using data stored in the storage node without requesting additional data from a different storage node, and send a computation result to the compute node;

a path-querying module, wherein in response to determining that executing the sub-task updates data locally stored on the storage node, the path-querying module is configured to query the master node to identify other storage nodes in the storage cluster that store replicas of the locally stored data; and

an update module configured to send the updated data to the identified other storage nodes.

2. The distributed computing system of claim 1 , wherein the master node comprises:

a receiving module configured to receive, from the compute node, compute context associated with to-be-written data;

a data-path generation module configured to generate data-placement paths based on the compute context; and

a transmitting module configured to provide the data-placement paths to the compute node to allow the compute node to write the to-be-written data to one or more storage nodes based on the data-placement paths.

3. The distributed computing system of claim 2 , wherein the compute node comprises:

a computation-task-partitioning module configured to partition the computation task into a number of sub-tasks based on the data-placement paths; and

a distribution module configured to send each sub-task to a corresponding single storage node.

4. The distributed computing system of claim 3 , wherein the compute node further comprises a result-gathering module configured to receive computation results from multiple storage nodes executing the sub-tasks to generate a combined result.

5. The distributed computing system of claim 2 , wherein the master node is configured to store the compute context as part of metadata of the to-be-written data.

6. The distributed computing system of claim 5 , wherein the update module of the storage node is further configured to:

in response to determining that executing the sub-task updates the locally stored data, send a data-update notification to the master node to allow the master node to update the corresponding metadata.

7. The distributed computing system of claim 1 , wherein the compute node comprises a first processor having a first processing speed, and wherein the storage node comprises a second processor having a second processing speed that is slower than the first processing speed.

8. A computer-implemented method for offloading computation tasks from a compute cluster comprising one or more compute nodes to a storage cluster comprising a plurality of storage nodes in a distributed computing system, the method comprising:

receiving, by a storage node from a compute node, computation instructions associated with a computation task, wherein the computation task is divided into a number of sub-tasks in such a way that each sub-task only requires data stored on a single storage node;

executing, by the storage node, a corresponding sub-task using data stored in the storage node without requesting additional data from a different storage node;

sending a computation result to the compute node;

in response to determining that executing the sub-task updates data locally stored on the storage node, querying a master node in the distributed computing system to identify other storage nodes in the storage cluster that store replicas of the data; and

sending the updated data to the identified other storage nodes.

9. The computer-implemented method of claim 8 , further comprising:

receiving, by the master node from the compute node, compute context associated with to-be-written data;

generating data-placement paths based on the compute context; and

providing the data-placement paths to the compute node to allow the compute node to write the to-be-written data to one or more storage nodes based on the data-placement paths.

10. The computer-implemented method of claim 9 , further comprising:

partitioning, by the compute node, the computation task into a number of sub-tasks based on the data-placement paths; and

sending each sub-task to a corresponding single storage node.

11. The computer-implemented method of claim 10 , further comprising receiving, by the compute node, computation results from multiple storage nodes executing the sub-tasks to generate a combined result.

12. The computer-implemented method of claim 9 , further comprising:

storing, by the master node, the compute context as part of metadata of the to-be-written data.

13. The computer-implemented method of claim 12 , further comprising:

in response to determining that executing the sub-task updates the locally stored data, sending, by the storage node, a data-update notification to the master node to allow the master node to update the corresponding metadata.

14. The computer-implemented method of claim 8 , wherein the compute node comprises a first processor having a first processing speed, and wherein the storage node comprises a second processor having a second processing speed that is lower than the first processing speed.

15. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for offloading computation tasks from a compute cluster comprising one or more compute nodes to a storage cluster comprising a plurality of storage nodes in a distributed computing system, the method comprising:

receiving, by a storage node from a compute node, computation instructions associated with a computation task, wherein the computation task is divided into a number of sub-tasks in such a way that each sub-task only requires data stored on a single storage node;

executing, by the storage node, a corresponding sub-task using data stored in the storage node without requesting additional data from a different storage node;

sending a computation result to the compute node;

in response to determining that executing the sub-task updates data locally stored on the storage node, querying a master node in the distributed computing system to identify other storage nodes in the storage cluster that store replicas of the data; and

sending the updated data to the identified other storage nodes.

16. The non-transitory computer-readable storage medium of claim 15 , wherein the method further comprises:

receiving, by the master node from the compute node, compute context associated with to-be-written data;

generating data-placement paths based on the compute context; and

providing the data-placement paths to the compute node to allow the compute node to write the to-be-written data to one or more storage nodes.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the method further comprises:

partitioning, by the compute node, the computation task into a number of sub-tasks based on the data-placement paths; and

sending each sub-task to a corresponding single storage node.

18. The non-transitory computer-readable storage medium of claim 17 , wherein the method further comprises receiving, by the compute node, computation results from multiple storage nodes executing the sub-tasks to generate a combined result.

19. The non-transitory computer-readable storage medium of claim 16 , wherein the method further comprises:

storing, by the master node, the compute context as part of metadata of the to-be-written data.

20. The non-transitory computer-readable storage medium of claim 19 , wherein the method further comprises:

in response to determining that executing the sub-task updates the locally stored data, sending, by the storage node, a data-update notification to the master node to allow the master node to update the corresponding metadata.