System and method for defining normal operating regions and identifying anomalous behavior of units within a fleet, operating in a complex, dynamic environment
Monitoring dynamic units that operate in complex, dynamic environments, is provided in order to classify and track unit behavior over time. When domain knowledge is available, feature-based models may be used to capture the essential state information of the units. When domain knowledge is not available, raw data is relied upon to perform this task. By analyzing logs of event messages (without having access to their data dictionary), embodiments allow the identification of anomalies (novelties). Specifically, a Normalized Compression Distance (such as one based on Kolmogorov Complexity) may be applied to logs of event messages. By analyzing the similarity and differences of the event message logs, units are identified that did not experience any abnormality (and locate regions of normal operations) and units that departed from such regions.
1. A method to monitor dynamic units that operate in dynamic environments comprising:
collecting raw data to capture state information of the units;
using a processor to analyze the raw data by obtaining a set of normalized compression distance (NCD) of the raw data based on a Kolmogorov complexity to identify an anomaly in a unit
visualizing the set of NCD in a two dimensional space;
minimizing distortion of the visualization by using a Kruskal's Stress-1 projection;
wherein the raw data is obtained by tracking an aircraft flight, the method further comprising tracking additional aircraft flights, computing a set of NCD for each additional flight, projecting the set of NCD in a two dimensional space for viewing, and classifying flights that are potential anomalies.
2. The method of claim 1 , wherein the raw data comprises event message logs.
3. The method of claim 1 , further comprising generating a one dimensional plot from the two dimensional projection to detect a trend of an anomaly.
4. The method of claim 1 , further comprising detecting an epidemic, wherein an epidemic comprises sustained/increasing numbers of anomalies over time.
5. The method of claim 1 , wherein the raw data comprises FADEC messages associated with a fleet of aircraft.
6. The method of claim 1 , wherein the units comprise CT scanners.
7. The method of claim 1 , further comprising: capturing state information of the units when domain knowledge is available with a feature-based models; and analyzing the state information to identify anomalies.
8. The method of claim 7 , wherein identifying abnormalities and regions of normal operations associated with units comprises combining: featureless similarity based anomaly detection; feature-based similarity based anomaly detection; and statistical parametric analysis.
9. A method to monitor dynamic units that operate in dynamic environments comprising:
defining a virtual string of data for a normal operation of the units in which no anomalies occur;
collecting raw data to capture state information of the units;
using a processor to analyze the raw data by obtaining a set of normalized compression distance (NCD) of the raw data versus the virtual string of data for a normal operation and based on a Kolmogorov complexity to identify anomaly in a unit; and
visualizing the set of NCD in a two dimensional space;
wherein the raw data is obtained by tracking an aircraft flight, the method further comprising tracking additional aircraft flights, computing a set of NCD for each additional flight, projecting the set of NCD in a two dimensional space for viewing, and classifying flights that are potential anomalies.
10. The method of claim 9 , wherein the raw data comprises event message logs.
11. The method of claim 9 , wherein the virtual normal data is obtained from a centroid of many unit operations.
12. The method of claim 9 , further comprising performing unsupervised clustering of the set of NCD.
13. The method of claim 9 , wherein an epidemic comprises sustained/increasing numbers of anomalies over time.
14. The method of claim 9 , wherein the raw data comprises FADEC messages associated with a fleet of aircraft.
15. The method of claim 9 , wherein the units comprise CT scanners.
16. The method of claim 9 , wherein the units are part of a fleet that operates in dynamic environments.