Apparatus and method for data processing
Embodiments of the present disclosure provide a method and apparatus for data processing. The apparatus comprises: a dividing unit configured to divide, based on a predetermined policy, computation workload for data in a storage node into at least one sub-workload; and a dispatching unit configured to dispatch the at least one sub-workload to at least one of the storage node and a computing node that is physically separate from the storage node for execution of the at least one sub-workload with computing resources in the at least one of the storage node and the computing node.
1. An apparatus for data processing, comprising:
a dividing unit configured to divide, based on a predetermined policy, computation workload for data in a storage device in a storage node into at least one sub-workload for execution by one or more of the storage node and a computing node; and
a dispatching unit configured to dispatch at least a portion of the at least one sub-workload from the storage device to the storage node for execution of the at least one sub-workload with computing resources in the storage node, the storage node being an external storage system, and the computing node being physically separate from the storage node, wherein dispatching the at least a portion of the at least one sub-workload from the storage device to the storage node for execution includes moving the at least a portion of the at least one sub-workload from the storage device to the computing resources in the storage node and wherein dispatching the at least a portion of the at least one sub-workload from the storage device to the computing node for execution includes transmitting the at least a portion of the at least one sub-workload from the storage device to the computing node via a network when additional computing capability beyond a computing capability of the storage node is needed to execute the at least a portion of the at least one sub-workload, wherein the computing capability of the storage node is determined by estimating an amount of required computing resources for executing all or part of the at least one sub-workload, wherein estimating the amount of required computing resources includes determining whether an amount of remaining computing resources is greater than the amount of required computing resources after computing resources used for providing data services from the computing resources of the storage node are removed.
2. The apparatus according to claim 1 , wherein the predetermined policy indicates at least one of the following:
dividing the computation workload based on respective computing capabilities of the computing node and the storage node; and
dividing the computation workload based on a type of the computation workload.
3. The apparatus according to claim 2 , wherein the at least one sub-workload at least include a first sub-workload.
4. The apparatus according to claim 3 , wherein the dividing unit is configured to divide all of the computation workload into the first sub-workload if the predetermined policy indicates dividing the computation workload based on the respective computing capabilities of the computing node and the storage node; and
wherein the dispatching unit is further configured to dispatch the first sub-workload to the storage node in response to determining that the computing capability of the storage node is sufficient to execute the first sub-workload.
5. The apparatus according to claim 3 , wherein the dividing unit is configured to divide a part of the computation workload into the first sub-workload if the predetermined policy indicates dividing the computation workload based on the respective computing capabilities of the computing node and the storage node; and
wherein the dispatching unit is further configured to dispatch the first sub-workload to the storage node in response to determining that the computing capability of the storage node is sufficient to execute the first sub-workload.
6. The apparatus according to claim 5 , wherein the at least one sub-workload further includes a second sub-workload that is different from the first sub-workload, the second sub-workload including a part of the computation workload; and
wherein the dispatching unit is further configured to dispatch the second sub-workload to the computing node.
7. The apparatus according to claim 4 , wherein the dividing unit is configured to divide the computation workload into an input/output (I/O)-intensive sub-workload if the predetermined policy indicates dividing the computation workload based on the respective computing capabilities of the computing node and the storage node and based on the type of the computation workload; and
wherein the dispatching unit is configured to dispatch the I/O-intensive sub-workload to the storage node.
8. The apparatus according to claim 6 , wherein if the predetermined policy indicates dividing the computation workload based on the type of the computation workload, the first sub-workload includes an input/output I/O-intensive sub-workload; and the second sub-workload includes a computing-intensive sub-workload.
9. The apparatus according to claim 6 , wherein there is dependency between the first and second sub-workloads, the apparatus further comprising:
a control unit configured to control data exchange between the computing node and the storage node based on the dependency.
10. The apparatus according to claim 9 , wherein the control unit is further configured to:
schedule execution of the first and second sub-workloads based on the dependency.
11. A method for data processing, comprising:
dividing, based on a predetermined policy, computation workload for data in a storage device in a storage node into at least one sub-workload for execution by one or more of the storage node and a computing node; and
dispatching at least a portion of the at least one sub-workload to the storage node for execution of the at least one sub-workload from the storage device with computing resources in the storage node, the storage node being an external storage system, and the computing node being physically separate from the storage node, wherein dispatching the at least a portion of the at least one sub-workload from the storage device to the storage node for execution includes moving the at least a portion of the at least one sub-workload from the storage device to the computing resources in the storage node and wherein dispatching the at least a portion of the at least one sub-workload from the storage device to the computing node for execution includes transmitting the at least a portion of the at least one sub-workload from the storage device to the computing node via a network if additional computing capability beyond a computing capability of the storage node is needed to execute the at least a portion of the at least one sub-workload wherein the computing capability of the storage node is determined by estimating an amount of required computing resources for executing all or part of the at least one sub-workload, wherein estimating the amount of required computing resources includes determining whether an amount of remaining computing resources is greater than the amount of required computing resources after computing resources used for providing data services from the computing resources of the storage node are removed.
12. The method according to claim 11 , wherein the predetermined policy indicates at least one of the following:
dividing the computation workload based on respective computing capabilities of the computing node and the storage node; and
dividing the computation workload based on a type of the computation workload.
13. The method according to claim 12 , wherein the at least one sub-workload at least include a first sub-workload.
14. The method according to claim 13 , wherein dividing the computation workload into the at least one sub-workload comprises dividing all of the computation workload into the first sub-workload if the predetermined policy indicates dividing the computation workload based on the respective computing capabilities of the computing node and the storage node, and
wherein dispatching the at least one sub-workload to at least one of the storage node and the computing node comprises dispatching the first sub-workload to the storage node in response to determining that the computing capability of the storage node is sufficient to execute the first sub-workload.
15. The method according to claim 13 , wherein dividing the computation workload into the at least one sub-workload comprises dividing a part of the computation workload into the first sub-workload if the predetermined policy indicates dividing the computation workload based on the respective computing capabilities of the computing node and the storage node; and
wherein dispatching the at least one sub-workload to at least one of the storage node and the computing node comprises dispatching the first sub-workload to the storage node in response to determining that the computing capability of the storage node is sufficient to execute the first sub-workload.
16. The method according to claim 15 , wherein the at least one sub-workload further includes a second sub-workload that is different from the first sub-workload, the second sub-workload including a part of the computation workload; and
wherein dispatching the at least one sub-workload to at least one of the storage node and the computing node comprises dispatching the second sub-workload to the computing node.
17. The method according to claim 14 , wherein dividing the computation workload into the at least one sub-workload comprises dividing the computation workload into an input/output (I/O)-intensive sub-workload if the predetermined policy indicates dividing the computation workload based on the respective computing capabilities of the computing node and the computing capability of the storage node and based on the type of the computation workload; and
wherein dispatching the at least one sub-workload to at least one of the storage node and the computing node comprises dispatching the I/O-intensive sub-workload to the storage node.
18. The method according to claim 16 , wherein if the predetermined policy indicates dividing the computation workload based on the type of the computation workload, the first sub-workload includes an input/output (I/O)-intensive sub-workload, and the second sub-workload includes a computing-intensive sub-workload.
19. The method according to claim 16 , wherein there is dependency between the first and second sub-workloads, the method further comprising:
controlling data exchange between the computing node and the storage node based on the dependency.
20. The method according to claim 19 , further comprising:
scheduling execution of the first and second sub-workloads based on the dependency.