Systems and/or methods for event stream deviation detection
Certain example embodiments described herein relate to systems and/or methods for event stream deviation detection. More particularly, certain example embodiments described herein relate to maintaining short and long-term statistics of an incoming stream of event data. In certain example embodiments, a deviation is calculated based at least in part on the long-term and short-term statistics. The deviation may then be compared to a threshold value. In certain example embodiments, the estimations required for the statistics are done with Kernel Density Estimators (KDEs).
1. A deviation detection method for use with a processing system including at least one processor, the method comprising:
(a) receiving, at the processing system, at least one stream of event data from a machine that is generating events for the stream of event data, the event data including at least one attribute of performance of a task by the machine;
(b) calculating, via the at least one processor, a long-term value by using a first estimate of a long-term probability density function (PDF) of the at least one monitored attribute in the at least one stream of event data over a long-term time window of the stream of event data;
(c) calculating, via the at least one processor, a short-term value by using a second estimate of a short-term PDF of the at least one monitored attribute in the at least one stream of event data over a short-term time window that is of a shorter duration than the long-term time window;
(d) computing, via the at least one processor, a first difference value between the calculated long-term value and an ideal density distribution value;
(e) computing, via the at least one processor, a second difference value between the calculated short-term value and the ideal density distribution value;
(f) computing, via the at least one processor, a current deviation as a function of the first difference value and the second difference value;
(g) comparing the current deviation to a threshold value;
(h) in response to the comparison of the current deviation exceeding the threshold value, triggering, in real-time with reception of the event stream, an adjustment in performance of the task being performed by the machine; and
repeating (a)-(h) as further monitored events are generated by the machine for performance of the task and provided by the at least one stream of event data.
2. The method of claim 1 , wherein the ideal density distribution of the stream of event data is based on multiple ideal values.
3. The method of claim 1 , wherein the short-term value is based on a short-term Kernel Density Estimator (KDE) and the long-term value is based on a long-term KDE.
4. The method of claim 1 , further comprising updating the long-term value with respect to a complete stream.
5. The method of claim 1 , wherein the threshold value is specified by a user.
6. The method of claim 1 , wherein the threshold value is dynamically determined by the processing system.
7. The method of claim 1 , wherein the calculating of the long-term value includes using Cluster Kernels.
8. The method of claim 1 , further comprising displaying a result of the comparing of the current deviation to the threshold value on a display in communication with the processing system.
9. The method of claim 1 , further comprising outputting a result of the comparing of the current deviation to the threshold value to a data store or external application.
10. The method of claim 1 , further comprising issuing a notification when the current deviation exceeds the threshold value.
11. The method of claim 1 , further comprising updating the long-term value with summaries of re-sampled partitions of the stream.
12. A deviation detection method for use with a processing system including at least one processor, the method comprising:
receiving at least one stream of event data at the processing system and from a machine that is generating events for the stream of event data, the event data including at least one attribute of performance of a task by the machine;
calculating, over a short-term time window, a short-term kernel density estimator (KDE) for at least one monitored event in the at least one stream of event data;
calculating, over a long-term time window that is longer than the first time period, a long-term KDE for the at least one monitored event in the at least one stream of event data;
computing, via the at least one processor, a first difference value as a function of the long-term KDE and an ideal density distribution value;
computing, via the at least one processor, a second difference value as a function of the short-term KDE and the ideal density distribution value;
calculating a deviation value as a function of the first difference value and the second difference value; and
comparing the deviation to a threshold to detect a deviation in the at least one stream of event data;
in response to the comparison of the current deviation exceeding the threshold value, triggering, in real-time with reception of the event stream, an adjustment in performance of the task being performed by the machine.
13. A deviation detection system for event processing, comprising:
an adapter configured to receive at least one stream of event data from a machine that is generating events for the stream of event data, the event data including at least one attribute of performance of a task by the machine;
at least one processor configured to:
calculate a long-term value by using a first estimate of a long-term probability density function (PDF) of at least one monitored attribute in the at least one stream of event data over a long-term time window of the stream of event data;
calculate a short-term value by using a second estimate of a short-term PDF of the at least one monitored attribute in the at least one stream of event data over a short-term time window that is of a shorter duration than the first time window;
compute a first difference value between the calculated long-term value and an ideal density distribution;
compute a second difference value between the calculated short-term value and the ideal density distribution value;
compute a current deviation as a function of the first difference value and the second difference value;
compare the current deviation to a threshold value;
in response to the comparison of the current deviation exceeding the threshold value, trigger, in real-time with reception of the event stream, an adjustment in performance of the task being performed by the machine; and
repeat the calculating, computing, and comparing as further monitored events are generated by the machine for performance of the task and received in the at least one stream of event data.
14. The system of claim 13 , wherein the short-term value is based on a short-term Kernel Density Estimator (KDE) and the long-term value is based on a long-term KDE.
15. The system of claim 13 , wherein the processor is configured to update the long-term value with respect to a complete stream.
16. The system of claim 13 , wherein the threshold value is specified by a user.
17. The system of claim 13 , wherein the threshold value is dynamically determined by the processing system.
18. The system of claim 13 , wherein the calculating of the long-term value includes using Cluster Kernels.
19. The system of claim 13 , further comprising a user interface configured to display a result of the comparing of the current deviation to the threshold.
20. The system of claim 13 , further comprising a data store configured to log information concerning detected deviations.
21. The system of claim 13 , wherein the processor is configured to update the long-term value with summaries of re-sampled partitions of the stream.
22. The system of claim 13 , wherein the computed current deviation is a difference between the first and second distances.
23. A non-transitory computer readable storage medium tangibly storing instructions that, when processed by at least one processor, executing a method according to claim 1 .