System and method for trend estimation for application-agnostic statistical fault detection
Trend estimation for application-agnostic statistical fault detection of multi-process applications in environments with data trend includes at least one of: a multi-process application runs on a host. Statistical events are collected and sent to a statistical fault detector. The statistical fault detector creates one or more distributions and compares recent statistical event data to historical statistical event data and uses deviation from historical norm for fault detection. Trend is estimated, and if needed, removed from event data prior to the creation of distributions. Trend is estimated using spectral techniques, filter banks and Maximum Entry Spectral Estimation, and dominant frequencies are estimated and utilized to adapt to the environment.
1. A system, comprising:
one or more memory locations configured to store one or more applications and one or more distributions, wherein each of said one or more applications is comprised of one or more processes and threads;
one or more Central Processing Units operatively connected to said one or more memory locations, configured to execute said one or more applications on a host with a host operating system, configured to generate one or more statistical events for said one or more executing applications and said host operating system, and configured to calculate said one or more distributions for said one or more statistical events;
one or more second memory locations configured to store one or more trend-estimators configured to estimate trends in said statistical events using one or more of a dominant period, a fixed number of events or a dynamic number of events, configured to calculate de-trended statistical event data, and configured to detect trends in the statistical events, wherein said one or more trend-estimators estimate trend using one or more of differencing, a moving average, and a least squares estimate and said trend-estimators are calculated using one or more of a number of elements equal to the dominant period, a period of one-half the dominant period, or a period less than one-half the dominant period; and
a fault detector configured to de-trend the data for said one or more statistical events prior to calculation of said one or more distributions if said one or more trend-estimator indicate trend.
2. The system according to claim 1 , wherein said operating system is one of Linux, UNIX, Solaris, HP/UX, Android, iOS, MacOS, or Microsoft Windows.
3. The system according to claim 1 , wherein said one or more distributions are calculated using a number of data elements that is one of the dominant period, one-half the dominant period, and a period less than one-half the dominant period.
4. The system according to claim 3 , wherein said moving average is one of simple moving average, exponential moving average, weighted moving average, and moving median.
5. The system according to claim 1 , wherein said one or more trend-estimators use one of the same number of data elements or a different number of data elements.
6. A system, comprising:
one or more memory locations configured to store one or more applications and one or more distributions, wherein each of said one or more applications is comprised of one or more processes and threads;
one or more Central Processing Units operatively connected to said one or more memory locations, configured to execute said one or more applications on a host with a host operating system, configured to generate one or more statistical events for said one or more executing applications and said host operating system, and configured to calculate said one or more distributions for said one or more statistical events;
one or more second memory locations configured to store one or more trend-estimators configured to estimate trends in said statistical events using one or more of a dominant period, a fixed number of events or a dynamic number of events, configured to calculate de-trended statistical event data, and configured to detect trends in the statistical events, wherein said dominant period is estimated using one or more of Fourier Transformation, Fast Fourier Transformation, Hilbert Transformation, filter bank, division of spectrum into frequency bands, or Maximum Entropy Spectral Estimation (MESE); and
a fault detector configured to de-trend the data for said one or more statistical events prior to calculation of said one or more distributions if said one or more trend-estimator indicate trend.
7. The system according to claim 6 , further comprising a trend-estimator configured to estimate a dominant frequency by division of the spectrum into equal sized frequency bands, configured to determine the frequency band with the highest amplitude, and configured to estimate the dominant frequency to be the center of said frequency band with the highest amplitude.
8. The system according to claim 6 , further comprising:
a trend-estimator configured to estimate a dominant frequency by the division of the spectrum into a number of frequency bands, configured to estimate the energy at the center of each band as the size of said band times the amplitude at said center frequency, configured to determine the frequency band with the highest energy, and configured to estimate the dominant frequency to be the center of said frequency band with the highest energy.
9. The system according to claim 6 , further comprising a trend-estimator configured to estimate a dominant frequency by the approximation of the spectrum by a continuous or continuous and smooth series of polynomials and configured to calculate the amplitude of said series of polynomials equidistantly across the entire spectrum, and configured to estimate the dominant frequency to be a frequency corresponding to the highest amplitude in said equidistantly calculated polynomials.
10. The system according to claim 6 , wherein said one or more trend-estimators use one of the same number of data elements or a different number of data elements.
11. The system according to claim 6 , wherein said filter bank is comprised of one or more band-pass filters.
12. The system according to claim 6 , further comprising a trend-estimator configured to estimate a dominant frequency by the calculation of the energy in each filter-bank, and configured to estimate the dominant frequency to be the mid-point frequency of the filter bank with the highest energy.
13. The system according to claim 12 , wherein the energy in each filter-bank is estimated to be the sum of the squared filtered event data over a number of filtered event data values.
14. The system according to claim 12 , wherein the energy in each filter-bank is estimated to be the sum of the squared filtered event data over filtered event data values for a period of time.
15. The system according to claim 6 , wherein said trend-estimators use a number of data elements that is proportional to the dominant period.
16. The system according to claim 6 , wherein said Maximum Entropy Spectral Estimate is calculated using Burgs algorithm.
17. The system according to claim 6 , further comprising a trend-estimator configured to estimate a dominant frequency by division of the Maximum Entropy Spectral Estimate spectrum into even sized frequency bands, configured to determine the frequency band with the highest amplitude, and configured to estimate the dominant frequency to be the center of said frequency band with the highest amplitude.
18. The system according to claim 6 , further comprising a trend-estimator configured to estimate a dominant frequency by the division of the Maximum Entropy Spectral Estimate spectrum into a number of frequency bands, configured to estimate the energy at the center of each band as the size of said band times the amplitude at said center frequency, configured to determine the frequency band with the highest energy, and configured to estimate the dominant frequency to be the center of said frequency band with the highest energy.
19. The system according to claim 6 , further comprising the one or more trend-estimators using one of a period of one-half the dominant period, and a period less than one-half the dominant period to estimate trend.
20. The system according to claim 6 , wherein said one or more distributions are calculated using a number of data elements that is one of the dominant period, one-half the dominant period, and a period less than one-half the dominant period.