Anomaly detection based on information technology environment topology
Techniques are described for analyzing data regarding activity in an IT environment to determine information regarding the entities associated with the activity and using the information to detect anomalous activity that may be indicative of malicious activity. In an embodiment, a plurality of events reflecting activity by a plurality of entities in an IT environment are processed to resolve the identities of the entities, discover how the entities fit within a topology of the IT environment, and determine what the entities are. This information is then used to generate an entity relationship graph that includes nodes representing the entities in the IT environment and edges connecting the nodes representing interaction relationships between the entities. In some embodiments, baselines are established by monitoring the activity between entities. This baseline information can be represented in the entity relationship graph in the form of directionality applied to the edges. The entity relationship graph can then be monitored to detect anomalous activity.
1. A computer implemented method comprising:
accessing a set of events associated with activity by a plurality of entities in an information technology (IT) environment, wherein each event in the set of events includes a portion of raw machine data that reflects activity in the IT environment and that is produced by a component of the IT environment, wherein each event is associated with a timestamp extracted from the raw machine data;
determining a topology of the IT environment by processing at least some of the accessed set of events;
generating an entity relationship graph based on the topology of the IT environment;
wherein the entity relationship graph includes:
a plurality of nodes representative of the plurality of entities in the IT environment; and
edges connecting the plurality of nodes, the edges representing relationships and activity between entities represented by the plurality of nodes;
wherein each edge includes a directionality that indicates a normal flow of communication between the entities represented by the nodes connected to the edge; and
monitoring the entity relationship graph to detect an anomaly.
2. The method of claim 1 , wherein the anomaly is detected in response to detecting a change in the entity relationship graph.
3. The method of claim 1 , wherein the anomaly is detected in response to detecting a shift in the directionality of an edge in the entity relationship graph.
4. The method of claim 1 , wherein the anomaly is indicative of anomalous communication between a particular entity of the plurality of entities and the at least one other entity of the plurality of entities.
5. The method of claim 1 , wherein the anomaly is indicative of a web shell attack.
6. The method of claim 1 , wherein monitoring the entity relationship graph includes:
focusing monitoring on a portion of the entity relationship graph associated with a particular logical location in the topology of the IT environment.
7. The method of claim 1 , further comprising:
outputting, via a user interface, an indication of the detected anomaly to a user.
8. The method of claim 1 , wherein the anomaly is detected based on detecting that the directionality has changed in at least one edge.
9. The method of claim 1 , wherein the anomaly is detected in response to identifying a communication between entities that does not conform with a directionality of an edge connecting nodes associated with the entities.
10. The method of claim 1 , further comprising:
updating the entity relationship graph as additional events are accessed and processed.
11. The method of claim 1 , further comprising:
associating an identifier to a particular entity of the plurality of the plurality of entities, the identifier extracted from at least some of the set events;
wherein the identifier includes any one or more of: a domain name, a uniform resource locater (URL), uniform resource identifier (URI), a unique identifier (UID), an Internet Protocol (IP) address, a Media Access Control (MAC) address, a device identification, or a user identification.
12. The method of claim 1 , further comprising:
extracting a plurality of identifiers from at least some of the accessed set of events; and
associating the plurality of identifiers to a particular entity of the plurality of entities.
13. The method of claim 1 , further comprising:
updating an identity resolution state table in real time as the set of events are accessed, the identity resolution state table associating a plurality of identifiers to a particular entity of the plurality of entities, the plurality of identifiers extracted from at least some of the accessed set of events.
14. The method of claim 1 , wherein determining the topology of the IT environment by processing at least some of the accessed set of events includes:
inferring logical relationships between the plurality of entities based on the activity by the plurality of entities.
15. The method of claim 1 , wherein determining the topology of the IT environment by processing at least some of the accessed set of events includes:
determining a plurality entity classes based on the activity by the plurality of entities.
16. The method of claim 1 , wherein determining the topology of the IT environment by processing at least some of the accessed set of events includes:
inferring a logical location of a particular entity of the plurality of entities in the IT environment based on activity by the particular entity;
wherein the logical location of the particular entity is any one of the logical locations from a set of logical locations including:
local area network (LAN);
demilitarized zone (DMZ);
wide area network (WAN); or
external.
17. The method of claim 1 , wherein determining the topology of the IT environment by processing at least some of the accessed set of events includes:
applying a topology label to an identifier referencing a particular entity of the plurality of entities, the topology label indicative of the location of the particular entity in the IT environment;
wherein the logical location of the particular entity is any one of:
local area network (LAN);
demilitarized zone (DMZ);
wide area network (WAN); or
external.
18. The method of claim 1 , further comprising:
receiving a user input defining a location of a particular entity in the IT environment;
applying a topology label to an identifier referencing the particular entity based on the user input; and
updating the topology of the IT environment based on the topology label.
19. The method of claim 1 , further comprising:
updating the topology of the IT environment as additional events are accessed and processed.
20. The method of claim 1 , further comprising:
outputting, via a user interface, information associated with the topology of the IT environment to a user.
21. The method of claim 1 , further comprising:
associating a particular entity of the plurality of entities with one of a plurality of entity classes
wherein the plurality of entity classes are predefined, user-defined, or defined based on processing of at least some of the events using supervised and/or unsupervised machine learning classification models.
22. The method of claim 1 , wherein the entity relationship graph is further based on behavioral profiles for one or more of the plurality of entities.
23. The method of claim 1 , further comprising:
generating a histogram based on activity by a particular entity of the plurality of entities;
comparing the histogram based on activity by the particular entity with a histogram based on activity by a plurality of entities associated with a particular class of entity; and
associating the particular entity with the particular class of entities if, based on the comparison, a matching criterion is satisfied.
24. The method of claim 1 , further comprising:
determining if a particular entity of the plurality of entities is operating as a client or a server relative to at least one other entity of the plurality of entities.
25. The method of claim 1 , wherein the set of events are accessed from a field-searchable data store, wherein a field is defined by an extraction rule or regular expression for extracting a subportion of text from the portion of raw machine data in an event to produce a value for the field for that event.
26. The method of claim 1 , wherein the plurality of entities include any of:
a device;
an application;
a user; or
data.
27. The method of claim 1 , wherein the events are received from a plurality of sources via an extract, transform, and load (ETL) pipeline.
28. The method of claim 1 , wherein the anomaly is detected in real time as events are accessed.
29. A computer system comprising:
a processor; and
a storage device having instructions stored thereon, which when executed by the processor cause the computer system to:
access a set of events associated with activity by a plurality of entities in an information technology (IT) environment, wherein each event in the set of events includes a portion of raw machine data that reflects activity in the IT environment and that is produced by a component of the IT environment, wherein each event is associated with a timestamp extracted from the raw machine data;
determine a topology of the IT environment by processing at least some of the accessed set of events;
generate an entity relationship graph based on the topology of the IT environment;
wherein the entity relationship graph includes:
a plurality of nodes representative of the plurality of entities in the IT environment; and
edges connecting the plurality of nodes, the edges representing relationships and activity between entities represented by the plurality of nodes;
wherein each edge includes a directionality that indicates a normal flow of communication between the entities represented by the nodes connected to the edge; and
monitor the entity relationship graph to detect an anomaly.
30. A non-transitory computer-readable medium containing instructions, execution of which in a computer system causes the computer system to:
access a set of events associated with activity by a plurality of entities in an information technology (IT) environment, wherein each event in the set of events includes a portion of raw machine data that reflects activity in the IT environment and that is produced by a component of the IT environment, wherein each event is associated with a timestamp extracted from the raw machine data;
determine a topology of the IT environment by processing at least some of the accessed set of events;
generate an entity relationship graph based on the topology of the IT environment;
wherein the entity relationship graph includes:
a plurality of nodes representative of the plurality of entities in the IT environment; and
edges connecting the plurality of nodes, the edges representing relationships and activity between entities represented by the plurality of nodes;
wherein each edge includes a directionality that indicates a normal flow of communication between the entities represented by the nodes connected to the edge; and
monitor the entity relationship graph to detect an anomaly.