Detecting and correcting a failure sequence in a computer system before a failure occurs

1. A method for detecting a failure sequence or other undesirable system behavior in a computer system and subsequently taking a corresponding remedial action, comprising:

receiving instrumentation signals from the computer system while the computer system is operating;

determining from the instrumentation signals if the computer system is in a failure sequence that is likely to lead to undesirable system behavior, such as a system crash, wherein determining if the computer system is in a failure sequence involves:

determining correlations between instrumentation signals in the computer system, wherein determining the correlations involves using a non-linear, non-parametric regression technique to determine the correlations, whereby the correlations can subsequently be used to generate estimated signals,

deriving estimated signals for a number of instrumentation signals, wherein each estimated signal is derived from correlations with other instrumentation signals, and

comparing an actual signal with an estimated signal for a number of instrumentation signal to determine whether the computer system is in a failure sequence;

wherein the determination involves considering predetermined multivariate correlations between multiple instrumentation signals and a failure sequence that is likely to lead to undesirable system behavior; and

if the computer system is in a failure sequence that is likely to lead to undesirable system behavior, taking a remedial action.

2. The method of claim 1 , wherein taking the remedial action involves generating an alarm.

3. The method of claim 2 , wherein generating the alarm involves communicating the alarm to a system administrator so that the system administrator can take the remedial action.

4. The method of claim 3 , wherein communicating the alarm to the system administrator involves communicating information specifying the nature of the failure sequence to the system administrator.

5. The method of claim 1 , wherein taking the remedial action can involve: killing processes, blocking creation of new processes, or throwing away work, until the system is no longer in a failure sequence that is likely to lead to undesirable system behavior.

6. The method of claim 1 , wherein comparing an actual signal with an estimated signal involves using sequential detection methods to detect changes in a relationship between the actual signal and the estimated signal.

7. The method of claim 6 , wherein the sequential detection methods include the Sequential Probability Ratio Test (SPRT).

8. The method of claim 1 , wherein determining the correlations involves:

deliberately overloading the computer system during a test mode to produce undesirable system behavior, such as a system crash; and

identifying multivariate correlations between multiple instrumentation signals and the system crash.

9. The method of claim 1 , wherein the non-linear, non-parametric regression technique can include a multivariate state estimation technique.

10. The method of claim 1 , wherein the instrumentation signals can include:

signals associated with internal performance parameters maintained by software within the computer system;

signals associated with physical performance parameters measured through sensors the computer system; and

signals associated with canary performance parameters for synthetic user transactions, which are periodically generated for performance measuring purposes.

11. A computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for detecting a failure sequence or other undesirable system behavior in a computer system and subsequently taking a corresponding remedial action, wherein the computer-readable storage medium includes magnetic and optical storage devices, disk drives, magnetic tape, CDs (compact discs), and DVDs (digital versatile discs or digital video discs), the method comprising: the method comprising:

receiving instrumentation signals from the computer system while the computer system is operating;

determining from the instrumentation signals if the computer system is in a failure sequence that is likely to lead to undesirable system behavior, such as a system crash; wherein determining if the computer system is in a failure sequence involves:

determining correlations between instrumentation signals in the computer system, wherein determining the correlations involves using a non-linear, non-parametric regression technique to determine the correlations, whereby the correlations can subsequently be used to generate estimated signals,

deriving estimated signals for a number of instrumentation signals, wherein each estimated signal is derived from correlations with other instrumentation signals, and

comparing an actual signal with an estimated signal for a number of instrumentation signal to determine whether the computer system is in a failure sequence;

wherein the determination involves considering predetermined multivariate correlations between multiple instrumentation signals and a failure sequence that is likely to lead to undesirable system behavior; and

if the computer system is in a failure sequence that is likely to lead to undesirable system behavior, taking a remedial action.

12. The computer-readable storage medium of claim 11 , wherein taking the remedial action involves generating an alarm.

13. The computer-readable storage medium of claim 12 , wherein generating the alarm involves communicating the alarm to a system administrator so that the system administrator can take the remedial action.

14. The computer-readable storage medium of claim 13 , wherein communicating the alarm to the system administrator involves communicating information specifying the nature of the failure sequence to the system administrator.

15. The computer-readable storage medium of claim 11 , wherein taking the remedial action can involve: killing processes, blocking creation of new processes, or throwing away work, until the system is no longer in a failure sequence that is likely to lead to undesirable system behavior.

16. The computer-readable storage medium of claim 11 , wherein comparing an actual signal with an estimated signal involves using sequential detection methods to detect changes in a relationship between the actual signal and the estimated signal.

17. The computer-readable storage medium of claim 16 , wherein the sequential detection methods include the Sequential Probability Ratio Test (SPRT).

18. The computer-readable storage medium of claim 11 , wherein determining the correlations involves:

deliberately overloading the computer system during a test mode to produce undesirable system behavior, such as a system crash; and

identifying multivariate correlations between multiple instrumentation signals and the system crash.

19. The computer-readable storage medium of claim 11 , wherein the non-linear, non-parametric regression technique can include a multivariate state estimation technique.

20. The computer-readable storage medium of claim 11 , wherein the instrumentation signals can include:

signals associated with internal performance parameters maintained by software within the computer system;

signals associated with physical performance parameters measured through sensors the computer system; and

signals associated with canary performance parameters for synthetic user transactions, which are periodically generated for performance measuring purposes.

21. An apparatus that detects a failure sequence or other undesirable system behavior in a computer system and subsequently takes a corresponding remedial action, comprising:

a monitoring mechanism configured to monitor instrumentation signals from the computer system while the computer system is operating;

a determination mechanism configured to determine from the instrumentation signals if the computer system is in a failure sequence that is likely to lead to undesirable system behavior, such as a system crash, wherein determining if the computer system is in a failure sequence involves:

determining correlations between instrumentation signals in the computer system, wherein determining the correlations involves using a non-linear, non-parametric regression technique to determine the correlations, whereby the correlations can subsequently be used to generate estimated signals,

deriving estimated signals for a number of instrumentation signals, wherein each estimated signal is derived from correlations with other instrumentation signals, and

comparing an actual signal with an estimated signal for a number of instrumentation signal to determine whether the computer system is in a failure sequence;

wherein the determination mechanism is based on multivariate correlations between multiple instrumentation signals and a failure sequence that is likely to lead to undesirable system behavior; and

a remediation mechanism that is configured to take a remedial action if the computer system is in a failure sequence that is likely to lead to undesirable system behavior.