Enhancing throughput and fault-tolerance in a parallel-processing system
One embodiment of the present invention provides a system that enhances throughput and fault-tolerance in a parallel-processing system. During operation, the system first receives a task. Next, the system partitions N computing nodes into M set-aside nodes and N-M primary computing nodes, wherein M≧1. The system then processes the task in parallel across the N-M primary computing nodes. While doing so, the system proactively monitors the health of each of the N-M primary computing nodes. If the system detects a node in the N-M primary computing nodes to be at risk of failure, the system copies the portion of the task associated with the at-risk node to a subset of the M set-aside nodes. The system then processes the portion of the task in parallel across the subset of the M set-aside nodes while the N-M primary computing nodes continue executing.
1. A method for enhancing throughput and fault-tolerance in a parallel-processing system, wherein the parallel-processing system comprises N computing nodes, the method comprising:
receiving a task;
partitioning the N computing nodes into M set-aside nodes and N-M primary computing nodes, wherein M≧1; and
processing the task in parallel across the N-M primary computing nodes by,
proactively monitoring the health of each of the N-M primary computing nodes; and
if a node in the N-M primary computing nodes is detected to be at risk of failure,
copying the portion of the task associated with the at-risk node to a subset of the M set-aside nodes; and
processing the portion of the task in parallel across the subset of the M set-aside nodes while the N-M primary computing nodes continue executing.
2. The method of claim 1 , wherein processing the task in parallel across the N-M primary computing nodes involves:
dividing the task into N-M subtasks; and
executing each of the N-M subtasks on one of the N-M primary computing nodes.
3. The method of claim 1 , wherein prior to copying the portion of the task associated with the at-risk node to the subset of the M set-aside nodes, the method further comprises:
proactively monitoring the health of each of the M set-aside nodes while the task is being processed by the N-M primary computing nodes; and
if a node in the M set-aside nodes is detected to be at risk of failure,
removing the node from the set-aside nodes; or
replacing the node with a healthy node.
4. The method of claim 3 , wherein monitoring the set-aside nodes involves running monitoring programs on the set-aside nodes.
5. The method of claim 1 , wherein the at-risk node is allowed to continue processing the associated portion of the task, after the portion of the task is copied to the subset of the M set-aside nodes.
6. The method of claim 5 , wherein if the at-risk node fails before completing the associated portion of the task, the method further comprises obtaining results for the associated portion of the task by combining the execution results for the portion of the task from the subset of the M set-aside nodes.
7. The method of claim 5 , wherein if the at-risk node does not fail before completing the associated portion of the task, the method further comprises replacing or repairing the at-risk node prior to processing the next task.
8. The method of claim 1 , wherein monitoring proactively the health of the N-M primary computing nodes involves monitoring telemetry signals from each of the N-M primary computing nodes.
9. A computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for enhancing throughput and fault-tolerance in a parallel-processing system, wherein the parallel-processing system comprises N computing nodes, the method comprising:
receiving a task;
partitioning the N computing nodes into M set-aside nodes and N-M primary computing nodes, wherein M≧1; and
processing the task in parallel across the N-M primary computing nodes by,
proactively monitoring the health of each of the N-M primary computing nodes; and
if a node in the N-M primary computing nodes is detected to be at risk of failure,
copying the portion of the task associated with the at-risk node to a subset of the M set-aside nodes; and
processing the portion of the task in parallel across the subset of the M set-aside nodes while the N-M primary computing nodes continue executing.
10. The computer-readable storage medium of claim 9 , wherein processing the task in parallel across the N-M primary computing nodes involves:
dividing the task into N-M subtasks; and
executing each of the N-M subtasks on one of the N-M primary computing nodes.
11. The computer-readable storage medium of claim 9 , wherein prior to copying the portion of the task associated with the at-risk node to the subset of the M set-aside nodes, the method further comprises:
proactively monitoring the health of each of the M set-aside nodes while the task is being processed by the N-M primary computing nodes; and
if a node in the M set-aside nodes is detected to be at risk of failure,
removing the node from the set-aside nodes; or
replacing the node with a healthy node.
12. The computer-readable storage medium of claim 11 , wherein monitoring the set-aside nodes involves running monitoring programs on the set-aside nodes.
13. The computer-readable storage medium of claim 9 , wherein the at-risk node is allowed to continue processing the associated portion of the task, after the portion of the task is copied to the subset of the M set-aside nodes.
14. The computer-readable storage medium of claim 13 , wherein if the at-risk node fails before completing the associated portion of the task, the method further comprises obtaining results for the associated portion of the task by combining the execution results for the portion of the task from the subset of the M set-aside nodes.
15. The computer-readable storage medium of claim 13 , wherein if the at-risk node does not fail before completing the associated portion of the task, the method further comprises replacing or repairing the at-risk node prior to processing the next task.
16. The computer-readable storage medium of claim 9 , wherein monitoring proactively the health of the N-M primary computing nodes involves monitoring telemetry signals from each of the N-M primary computing nodes.
17. An apparatus that enhances throughput and fault-tolerance in a parallel-processing system, wherein the parallel-processing system comprises N computing nodes, comprising:
a receiving mechanism configured to receive a task;
a partitioning mechanism configured to partition the N computing nodes into M set-aside nodes and N-M primary computing nodes, wherein M≧1;
a monitoring mechanism configured to proactively monitor the health of each of the N-M primary computing nodes while the task is being processed in parallel across the N-M primary computing nodes; and
wherein if the monitoring mechanism detects a node in the N-M primary computing to be at risk of failure, the monitoring mechanism is configured to
copy the portion of the task associated with the at-risk node to a subset of the M set-aside nodes; and to
process the portion of the task in parallel across the subset of the M set-aside nodes while the N-M primary computing nodes continue executing.
18. The apparatus of claim 17 , further comprising:
a dividing mechanism configured to divide the task into N-M subtasks; and
an execution mechanism configured to execute each of the N-M subtasks on one of the N-M primary computing nodes.
19. The apparatus of claim 17 , further comprising:
a monitoring mechanism configured to proactively monitor the health of each of the M set-aside nodes while the task is being processed by the N-M primary computing nodes;
wherein if the monitoring mechanism detects a node in the M set-aside nodes to be at risk of failure, the monitoring mechanism is configured to
remove the node from the set-aside nodes; or to
replace the node with a healthy node.
20. The apparatus of claim 19 , wherein the monitoring mechanism is configured to run monitoring programs on the set-aside nodes.
21. The apparatus of claim 17 , wherein the at-risk node is allowed to continue processing the associated portion of the task after copying the portion of the task to the subset of the M set-aside nodes.