Anomaly detection in a data stream
There is provided a method for detecting an anomaly in plurality of data streams originating from a system or network of systems. Data streams are collected from the system or systems and divided into a plurality of time intervals. For each of the plurality of time intervals, a value for a parameter associated with the data stream is determined. A deviation in the determined values is calculated for the parameters associated with the data stream from expected values for the parameters and, if the calculated deviation is above a threshold, an anomaly is detected in the collected data stream.
1. An anomaly detection method comprising:
receiving in memory of a computing system, a data stream of data;
segmenting the data stream into different time intervals and computing one or more values for one or more respective sequences of events detected in the data stream of data for each of the time intervals in a corresponding segment, the computing comprising, for each one of the sequence of events, mapping a keyword contained therein to a specific severity value so as to produce a data series of severity values defining a graphically plottable stochastic severity function;
comparing each computed one of the severity values in the corresponding segment to an expected value for the corresponding segment comprising a median value for all data in the corresponding segment in order to determine a deviation value for the corresponding segment; and,
declaring an anomalous condition for the corresponding segment, computing a deviation function of a sequence of deviations determined in respect to the data stream, classifying the anomalous condition by comparing the sequence of deviations to different fingerprint sequences pre-stored in a data store, the fingerprint sequences each classifying a different respective anomaly through pattern recognition, in order to locate a similar one of the fingerprint sequences within a threshold value and applying a classification of the similar one of the fingerprint sequences to the sequence of deviations and assigning an action in the computer in response to the anomalous condition pre-associated with applied classification, the action assigned when the deviation value exceeds a threshold value for the corresponding segment.
2. The method of claim 1 , further comprising defining each of the time intervals as a function of a duration of time during which sequences of events are to be collected in order to produce a continuous or discrete function representing a sequence of events, and a radius of time, denoting if the anomalous condition may be declared within this radius of time.
3. The method of claim 1 , wherein the deviation function is a plot of a histogram of the deviations visually characterized by a unique shape and each of the fingerprint sequences are individual histograms each having a unique shape, the comparison of the sequence of deviations to the different fingerprint sequences pre-stored in a data store comprising a comparison of the unique shape of the histogram of the deviations to each different histogram of the fingerprint sequences so as to locate one of the fingerprint sequences with an associated histogram of similar shape as the histogram of the deviations.
4. The method of claim 1 , further comprising, when none of the fingerprint sequences in the data store are determined to be similar to the deviation function, adding the deviation function to the data store as a new one of the fingerprint sequences and associating the new one of the fingerprint sequences with an unknown classification.
5. An anomaly detection system comprising:
a computing system comprising a computer with memory and at least one processor; and,
an anomaly detector comprising computer program instructions executing in the memory of the computer during which execution the anomaly detector performs:
receiving in the memory of the computer, a data stream of data;
segmenting the data stream into different time intervals and computing one or more values for one or more respective sequences of events detected in the data stream of data for each of the time intervals in a corresponding segment, the computing comprising, for each one of the sequence of events, mapping a keyword contained therein to a specific severity value so as to produce a data series of severity values defining a graphically plottable stochastic severity function;
comparing each computed one of the severity values in the corresponding segment to an expected value for the corresponding segment comprising a median value for all data in the corresponding segment in order to determine a deviation value for the corresponding segment; and,
declaring an anomalous condition for the corresponding segment, computing a deviation function of a sequence of deviations determined in respect to the data stream, classifying the anomalous condition by comparing the sequence of deviations to different fingerprint sequences pre-stored in a data store, the fingerprint sequences each classifying a different respective anomaly through pattern recognition, in order to locate a similar one of the fingerprint sequences within a threshold value and applying a classification of the similar one of the fingerprint sequences to the sequence of deviations and assigning an action in the computer in response to the anomalous condition pre-associated with applied classification, the action assigned when the deviation value exceeds a threshold value for the corresponding segment.
6. The anomaly detection system of claim 5 , wherein the anomaly detector further performs defining each of the time intervals as a function of a duration of time during which sequences of events are to be collected in order to produce a continuous or discrete function representing a sequence of events, and a radius of time, denoting if the anomalous condition may be declared within this radius of time.
7. The anomaly detection system of claim 5 , wherein the deviation function is a plot of a histogram of the deviations visually characterized by a unique shape and each of the fingerprint sequences are individual histograms each having a unique shape, the comparison of the sequence of deviations to the different fingerprint sequences pre-stored in a data store comprising a comparison of the unique shape of the histogram of the deviations to each different histogram of the fingerprint sequences so as to locate one of the fingerprint sequences with an associated histogram of similar shape as the histogram of the deviations.
8. The anomaly detection system of claim 5 , wherein the anomaly detector further performs, when none of the fingerprint sequences in the data store are determined to be similar to the deviation function, adding the deviation function to the data store as a new one of the fingerprint sequences and associating the new one of the fingerprint sequences with an unknown classification.