Processing and storing event data in a knowledge graph format for anomaly detection
A technique for anomaly detection is disclosed. Event data is converted into a normalized common information model. The resulting data may be stored in an event data store database. Additionally, the resulting data may be stored in a knowledge graph representation in a knowledge graph database. The knowledge graph database efficiently stores event data to generate histograms on demand for common anomaly queries.
1. A computer-implemented method for storing data for anomaly detection, comprising:
monitoring event data in an information technology (IT) environment for at least one of machine users and human users;
mapping collected event data into a common information model;
inserting event data, mapped into the common information model, into an event store database;
transforming the event data, mapped into the common information model, into a knowledge graph representation in which events are represented by nodes and links between nodes, with the links including attributes of time series data for event activity strings; wherein the knowledge graph representation is selected to store event data to generate histograms on demand for a set of common anomaly queries and to serve as an index into the event store database; and
performing query processing using the knowledge graph representation as an index into event data in the event store database to answer a selected query.
2. The computer-implemented method of claim 1 , further comprising qualifying events in the knowledge graph representation for persistence.
3. The computer-implemented method of claim 1 , wherein monitoring event data of at least one of machine users and human users comprises monitoring at least one of: identity access and management tools, endpoint activity monitoring tools, cloud activity monitoring tools, email activity monitoring tools, network activity monitoring tools, SaaS infrastructure activity monitoring tools, managed device monitoring tools, and IoT monitoring tools.
4. The computer-implemented method of claim 1 , wherein the knowledge graph representation has a format that represents at least one time/occurrence activity data string in hexadecimal format.
5. The computer-implemented method of claim 4 , wherein the hexadecimal format utilizes a hexadecimal value to represent a single character for four quarters in a day.
6. The computer-implemented method of claim 1 , wherein the common information model provides standardized fields to represent event data from a plurality of different proprietary vendor formats.
7. A computer-implemented method for storing data for anomaly detection, comprising:
monitoring event data in an information technology (IT) environment for at least one of machine users and human users;
mapping collected event data into a common information model;
inserting event data, mapped into the common information model, into an event store database;
transforming the event data, mapped into the common information model, into a knowledge graph representation in which events are represented by nodes and links between nodes, with the links including attributes of time series data for event activity strings wherein the links include configuration filters for time series data indicative of one or more values of conditions associated with events, and wherein the knowledge graph representation is selected to store event data to generate histograms on demand for a set of common anomaly queries; and
performing query processing by using the knowledge graph representation to generate histograms on demand to answer a selected query.
8. The computer-implemented method of claim 7 , further comprising:
performing anomaly detection based on the generated histograms.
9. A computer-implemented method for storing data for anomaly detection, comprising:
monitoring event data in an information technology (IT) environment for at least one of machine users and human users;
mapping collected event data into a common information model;
inserting event data, mapped into the common information model, into an event store database;
transforming the event data, mapped into the common information model, into a knowledge graph representation in which events are represented by nodes and links between nodes, with the links including attributes of time series data for event activity strings wherein the links include configuration filters for time series data indicative of one or more values of conditions associated with events, and wherein the knowledge graph representation is selected to store event data to generate histograms on demand for a set of common anomaly queries; and
performing anomaly detection by decoding the knowledge graph representation to generate at least one histogram on demand and using the at least one histogram to identify an anomaly.
10. A computer-implemented method for storing data for anomaly detection, comprising:
monitoring event data in an information technology (IT) environment for at least one of machine users and human users;
mapping collected event data into a common information model;
inserting event data, mapped into the common information model, into an event store database;
transforming the event data, mapped into the common information model, into a knowledge graph representation in which events are represented by nodes and links between nodes, with the links including attributes of time series data for event activity strings, wherein the knowledge graph representation is selected to store event data to generate histograms on demand for a set of common anomaly queries and to serve as an index into the event data for anomaly queries not directly answerable from data in the knowledge graph representation;
performing query processing, including;
in response to a selected query being directly answerable from data in the knowledge graph representation, generating a histogram from the knowledge graph representation to answer the query; and
in response to the selected query not being directly answerable from the knowledge graph representation, using the knowledge graph representation as an index into the event store database to generate a histogram from the event data store to answer the selected query.
11. An anomaly detection system, comprising:
a processor and a memory configured to implement a method including:
monitoring event data in an information technology (IT) environment for at least one of machine users and human users;
mapping collected event data into a common information model;
inserting event data, mapped into the common information model, into an event store database;
transforming the event data, mapped into the common information model, into a knowledge graph representation in which events are represented by nodes and links between nodes, with the links including attributes of time series data for event activity strings; wherein the knowledge graph representation is selected to store event data to generate histograms on demand for a set of common anomaly queries and to serve as an index into the event store database; and
performing query processing using the knowledge graph representation as an index into event data in the event store database to answer a selected query.
12. The system of claim 11 , wherein the method further comprises qualifying events in the knowledge graph representation for persistence.
13. The system of claim 11 , wherein monitoring event data of at least one of machine users and human users comprises monitoring at least one of: identity access and management tools, endpoint activity monitoring tools, cloud activity monitoring tools, email activity monitoring tools, network activity monitoring tools, SaaS infrastructure activity monitoring tools, managed device monitoring tools, and IoT monitoring tools.
14. The system of claim 11 , wherein the knowledge graph representation has a format that represents at least one time/occurrence activity data string in hexadecimal format.
15. The system of claim 14 , wherein the hexadecimal format utilizes a hexadecimal value to represent a single character for four quarters in a day.
16. The system of claim 11 , wherein the common information model provides standardized fields to represent event data from a plurality of different proprietary vendor formats.
17. A system comprising:
a processor and a memory configured to implement a method including:
monitoring event data in an information technology (IT) environment for at least one of machine users and human users;
mapping collected event data into a common information model;
inserting event data, mapped into the common information model, into an event store database;
transforming the event data, mapped into the common information model, into a knowledge graph representation in which events are represented by nodes and links between nodes, with the links including attributes of time series data for event activity strings wherein the links include configuration filters for time series data indicative of one or more values of conditions associated with events, and wherein the knowledge graph representation is selected to store event data to generate histograms on demand for a set of common anomaly queries; and
performing query processing using the knowledge graph representation to generate histograms on demand to answer a selected query.
18. The system of claim 17 , wherein the method further comprises:
performing anomaly detection based on the generated histograms.
19. A system comprising:
a processor and a memory configured to implement a method including:
monitoring event data in an information technology (IT) environment for at least one of machine users and human users;
mapping collected event data into a common information model;
inserting event data, mapped into the common information model, into an event store database;
transforming the event data, mapped into the common information model, into a knowledge graph representation in which events are represented by nodes and links between nodes, with the links including attributes of time series data for event activity strings wherein the links include configuration filters for time series data indicative of one or more values of conditions associated with events, and wherein the knowledge graph representation is selected to store event data to generate histograms on demand for a set of common anomaly queries; and
performing anomaly detection by decoding the knowledge graph representation to generate at least one histogram on demand and using the at least one histogram to identify an anomaly.
20. A system comprising:
a processor and a memory configured to implement a method including:
monitoring event data in an information technology (IT) environment for at least one of machine users and human users;
mapping collected event data into a common information model;
inserting event data, mapped into the common information model, into an event store database;
transforming the event data, mapped into the common information model, into a knowledge graph representation in which events are represented by nodes and links between nodes, with the links including attributes of time series data for event activity strings, wherein the knowledge graph representation is selected to store event data to generate histograms on demand for a set of common anomaly queries and to serve as an index into the event data for anomaly queries not directly answerable from data in the knowledge graph representation;
performing query processing, including:
in response to a selected query being directly answerable from data in the knowledge graph representation, generating a histogram from the knowledge graph representation to answer the query; and
in response to the selected query not being directly answerable from the knowledge graph representation, using the knowledge graph representation as an index into the event store database to generate a histogram from the event data store to answer the selected query.