Processing pipeline for monitoring information systems
A processing pipeline for supporting advanced analytics for network monitoring and information management as well as specific analytics for particular use cases. The processing pipeline 500 takes in system data ( 502 ) and pre-processes ( 504 ) the system data. The system data ( 502 ) may include any of the types of data described above including text log files, and categorical data from various sources. The illustrated processing pipeline 500 includes two branches; a data modeling branch ( 506 ) where a model is developed for the data and a data evaluation branch ( 508 ) where the developed model is leveraged to evaluate live data. For certain event detection use cases, the output of the data evaluation branch ( 508 ) includes a score ( 510 ) (e.g., a threat level score) and context information for evaluating the threat.
1. A method for use in monitoring one or more platforms of one or more data systems, comprising:
receiving, at a processing system, system data generated by one or more source platforms and transmitted over at least one communications network;
first operating said processing system to prepare said system data for an analytics process so as to yield input data for said analytics process, said input data having one or more data features and observations for said data features, wherein said first processing comprises synthesizing at least one feature from said system data and the at least one feature enables processing to discriminate between benign activity and events of interest;
second operating said processing system to execute said analytics process with respect to said input data so as to yield output data, said analytics process involving at least one of data modeling, wherein a data model for said input data is developed, and data evaluation, wherein said data model is used to evaluate said input data; and
third operating said processing system to use said output data of said analytics process to provide information concerning at least one event of interest of the events of interest regarding said system data.
2. The method of claim 1 , wherein said first operating comprises cleaning said system data by filtering said structured data with respect to at least one of 1) data having too few observations regarding said attributes, and 2) data associated with a specified source or target platform.
3. The method of claim 1 , wherein said first operating comprises identifying a data object of said system data that is missing at least one value and processing said system data to address said missing value.
4. The method of claim 3 , wherein said processing said input data to address said missing value comprises one of inferring said missing value and assigning said missing value as unknown.
5. The method of claim 1 , wherein said first processing comprises enhancing said input data with supplementary data from an external source separate from said source platform.
6. The method of claim 1 , wherein said synthesizing involves identifying a feature space and mapping said system data to said feature space.
7. The method of claim 1 , wherein said synthesizing comprises vectorization of said system data into vectors each including N observations for each of P features.
8. The method of claim 6 , wherein each vector is aggregated over a defined time window of said structured data.
9. The method of claim 1 , wherein said synthesizing comprises converting said system data into tensors each including N observations for each of a set of multidimensional data.
10. The method of claim 1 , wherein said second operating comprises developing a model based on said system data.
11. The method of claim 9 , wherein said second operating comprises identifying an anomaly based on a deviation from said model based on said observations.
12. The method of claim 10 , wherein said second operating comprises quantifying a degree of said deviation from said model.
13. The method of claim 1 , wherein said third operating comprise providing a score for a first event of interest of the events of interest.
14. The method of claim 1 , wherein said third operating comprises normalizing and ranking a degree of security risk for each of first and second events of interest of the events of interest.
15. The method of claim 14 , wherein said ranking involves comparing first and second deviations from said baseline by aggregating deviation scores over a set of time windows of said system data.
16. The method of claim 14 , wherein said ranking involves normalizing data concerning at least one of said first and second deviations so as to facilitate comparisons therebetween.
17. An apparatus for use in monitoring one or more source platforms of one or more data systems, comprising:
a processing system operative to:
1) receive system data generated by one or more source platforms and transmitted over at least one communications network,
2) prepare said system data for an analytics process so as to yield input data for said analytics process, said input data having one or more data features and observations for said data features, wherein said first processing comprises synthesizing at least one feature from said system data and the at least one feature enables processing to discriminate between benign activity and events of interest,
3) execute said analytics process with respect to said input data so as to yield output data, said analytics process involving at least one of data modeling, wherein a data model for said input data is developed, and data evaluation, wherein said data model is used to evaluate said input data; and
4) use said output data of said analytics process to provide information concerning at least one event of interest of the events of interest of regarding said system data.
18. The apparatus of claim 17 , wherein said processing system is operative for cleaning said system data by filtering said structured data with respect to at least one of 1) data having too few observations regarding said attributes, and 2) data associated with a specified source or target platform.
19. The apparatus of claim 17 , wherein said processing system is operative for identifying a data object of said system data that is missing at least one value and processing said system data to address said missing value.
20. The apparatus of claim 19 , wherein said processing system is operative to address said missing value by one of inferring said missing value and assigning said missing value as unknown.
21. The apparatus of claim 17 , wherein said processing system is operative for enhancing said input data with supplementary data from an external source separate from said source platform.
22. The apparatus of claim 17 , wherein said processing system is operative for identifying a feature space and mapping said system data to said feature space.
23. The apparatus of claim 17 , wherein said processing system is operative for vectorization of said system data into vectors each including N observations for each of P features.
24. The apparatus of claim 17 , wherein each vector is aggregated over a defined time window of said structured data.
25. The apparatus of claim 17 , wherein said processing system is operative for converting said system data into tensors each including N observations for each of a set of multidimensional data.
26. The apparatus of claim 17 , wherein said processing system is operative for developing a model based on said system data.
27. The apparatus of claim 26 , wherein said processing system is operative for identifying an anomaly based on a deviation from said model based on said observations.
28. The apparatus of claim 27 , wherein said processing system is operative for quantifying a degree of said deviation from said model.
29. The apparatus of claim 17 , wherein said processing system is operative for providing a score for a first event of interest of the events of interest.
30. The apparatus of claim 17 , wherein said processing system is operative for normalizing and ranking a degree of security risk for each of first and second events of interest of the events of interest.
31. The apparatus of claim 30 , wherein said ranking involves comparing first and second deviations from said baseline by aggregating deviation scores over a set of time windows of said system data.
32. The apparatus of claim 30 , wherein said ranking involves normalizing data concerning at least one of said first and second deviations so as to facilitate comparisons therebetween.