Risk indices for enhanced throughput in computing systems
View Patent ↗Embodiments of a system that adjusts a checkpointing frequency in a distributed computing system that executes multiple jobs are described. During operation, the system receives signals associated with the operation of the computing nodes. Then, the system determines risk metrics for the computing nodes using a pattern-recognition technique to identify anomalous signals in the received signals. Next, the system adjusts a checkpointing frequency of a given checkpoint for a given computing node based on a comparison of a risk metric associated with the given computing node and a threshold, thereby implementing holistic fault tolerance, in which prediction and prevention of potential faults occurs across the distributed computing system.
1. A method for adjusting a checkpointing frequency in a distributed computing system that executes multiple jobs, comprising:
receiving signals associated with operation of the computing nodes;
determining risk metrics for the computing nodes using a pattern-recognition technique to identify anomalous signals in the received signals; and
adjusting a checkpointing frequency of a given checkpoint for a given computing node based on a comparison of a risk metric associated with the given computing node and a threshold, thereby implementing holistic fault tolerance, in which prediction and prevention of potential faults occurs across the distributed computing system.
2. The method of claim 1 , further comprising redirecting jobs to computing nodes having risk metrics that are less than the threshold.
3. The method of claim 1 , wherein the signals include telemetry signals measured in the computing system.
4. The method of claim 1 , wherein the signals are associated with different types of sensors in the computing system.
5. The method of claim 1 , wherein the risk metrics include alerts that are associated with degradation of components in the computing nodes.
6. The method of claim 1 , wherein the pattern-recognition technique includes a sequential probability ratio test (SPRT).
7. The method of claim 1 , wherein the pattern-recognition technique includes a nonlinear, nonparametric regression technique.
8. The method of claim 1 , wherein the threshold corresponds to a false-alarm frequency; and
wherein the false-alarm frequency is associated with an exponentially distributed random variable.
9. The method of claim 1 , wherein the anomalous signals are identified based on a slope of a cumulative function of the risk metric over time.
10. The method of claim 9 , wherein an amount of degradation of a component in the given computing node corresponds to a magnitude of change in the mean or standard deviation of the slope.
11. The method of claim 9 , wherein an amount of degradation of a component in the given computing node corresponds to a rate of change in the mean or standard deviation of the slope.
12. The method of claim 9 , wherein the slope is independent of units of one or more received signals that are used to determine the risk metric.
13. The method of claim 1 , wherein the given checkpoint is configured to interrupt operation of the given computing node and to store a current state of a job executing on the given computing node for use in subsequent recovery of the job.
14. The method of claim 1 , further comprising executing jobs on computing nodes having risk metrics that are less than the threshold on additional nodes.
15. A computer-program product for use in conjunction with a computer system, the computer-program product comprising a computer-readable storage medium and a computer-program mechanism embedded therein for adjusting a checkpointing frequency in a distributed computing system that executes multiple jobs on computing nodes, comprising:
instructions for receiving signals associated with operation of the computing nodes;
instructions for determining risk metrics for the computing nodes using a pattern-recognition technique to identify anomalous signals in the received signals; and
instructions for adjusting a checkpointing frequency of a given checkpoint for a given computing node based on a comparison of a risk metric associated with the given computing node and a threshold, thereby implementing holistic fault tolerance, in which prediction and prevention of potential faults occurs across the distributed computing system.
16. The computer-program product of claim 15 , further comprising instructions for redirecting jobs to computing nodes having risk metrics that are less than the threshold.
17. The computer-program product of claim 15 , wherein the risk metrics include alerts that are associated with degradation of components in the computing nodes.
18. The computer-program product of claim 15 , wherein the pattern-recognition technique includes a sequential probability ratio test (SPRT).
19. The computer-program product of claim 15 , wherein the pattern-recognition technique includes a nonlinear, nonparametric regression technique.
20. A distributed computer system, comprising:
multiple coupled computing nodes that execute multiple jobs;
multiple checkpoints, wherein a given checkpoint is associated with a given computing node;
memory; and
a program module, wherein the program module is stored in the memory and configured to be executed by a processor, which is included in at least one of the computing nodes, to adjust a checkpoint frequency of one or more checkpoints, the program module including:
instructions for receiving signals associated with operation of the computing nodes;
instructions for determining risk metrics for the computing nodes using a pattern-recognition technique to identify anomalous signals in the received signals; and
instructions for adjusting a checkpointing frequency of the given checkpoint based on a comparison of a risk metric and a threshold, thereby implementing holistic fault tolerance, in which prediction and prevention of potential faults occurs across the distributed computing system.