Identity resolution in data intake of a distributed data processing system
A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.
1. A computer-implemented method comprising:
identifying a plurality of entities involved in a plurality of events on a computer network, the plurality of entities including a particular user represented by a user identifier and a machine represented by a machine identifier;
determining that the machine identifier is to be associated with the particular user, based on a statistical analysis of events, of the plurality of events, that satisfy a criterion; and
in response to a determination that the machine identifier is to be associated with the particular user, annotating raw, unstructured machine data of a particular event to include an indication that the particular event is associated with the particular user.
2. The method of claim 1 , wherein the criterion comprises a probability of association between the particular user and the machine identifier exceeding a confidence threshold.
3. The method of claim 1 , wherein the annotating is performed regardless of whether the particular event includes the user identifier.
4. The method of claim 1 , wherein the annotating is performed when the particular event includes the machine identifier.
5. The method of claim 1 , wherein the annotating is performed when the particular event includes the machine identifier but not the user identifier.
6. The method of claim 1 , wherein the annotating is performed when the particular event is received during a valid time period.
7. The method of claim 1 , wherein said determining step comprises:
creating a probabilistic graph to generate and track a probability of association between the particular user and the machine identifier,
wherein a result from the probabilistic graph has a time-based dependence on current and past inputs.
8. The method of claim 1 , wherein said determining step comprises:
creating a probabilistic graph to record a probability of association between the particular user and the machine identifier,
wherein the probabilistic graph includes a peripheral node, a center node, and an edge, the peripheral node representing the machine identifier, the center node representing the particular user, and the edge representing the probability of association between the machine identifier and the particular user.
9. The method of claim 1 , wherein said determining step comprises:
creating a probabilistic graph to record a probability of association between the particular user and the machine identifier,
wherein the probabilistic graph is in the form of a stored data structure, and
wherein the stored data structure is configured to include additional machine identifiers.
10. The method of claim 1 , further comprising:
updating a probability of association between the particular user and the machine identifier upon receiving a new event having at least one of: the machine identifier or the user identifier.
11. The method of claim 1 , further comprising:
updating a probability of association between the particular user and the machine identifier upon receiving a new event having at least one of: the machine identifier or the user identifier;
wherein the new event comprises an authentication event that includes the user identifier.
12. The method of claim 1 , further comprising:
updating a probability of association between the particular user and the machine identifier upon receiving a new event having at least one of: the machine identifier or the user identifier;
wherein the new event comprises an authentication event that includes the user identifier, and
wherein said updating step adjusts a weight assigned to the new event based on a type of authentication event.
13. The method of claim 1 , further comprising:
updating a probability of association between the particular user and the machine identifier upon receiving a new event having at least one of: the machine identifier or the user identifier;
wherein the new event comprises an authentication event that includes the user identifier,
wherein said updating step assigns more weight to a physical login type of authentication event than to any other type of authentication event.
14. The method of claim 1 , further comprising:
creating, by a machine learning model, a probabilistic graph to record a probability of association between the particular user and the machine identifier.
15. The method of claim 1 , wherein said determining step is performed only on events that have occurred during a lifetime of a particular version of a machine learning model that is used to generate and track a probability of association between the particular user and the machine identifier.
16. The method of claim 1 , wherein the plurality of events is received in an order different from a temporal order of the events.
17. The method of claim 1 , further comprising:
sending a user association record to a cache server.
18. The method of claim 1 , further comprising:
sending a user association record to a cache server that stores structured data,
wherein the user association record is stored in the cache server using a data structure representing a probability of association between the particular user and each of a plurality of machine identifiers.
19. The method of claim 1 , wherein the events further include a second machine identifier, the method further comprising:
determining a probability of association between the machine identifier and the second machine identifier.
20. The method of claim 1 , wherein the events further include a second machine identifier, the method further comprising:
determining a probability of machine association between the machine identifier and the second machine identifier; and
upon the probability of machine association satisfying a second specified criterion, creating a machine association record indicative that a particular event having the second machine identifier is associated with the machine identifier.
21. The method of claim 1 , further comprising:
resolving a user identity of the particular user by querying, using the user identifier as a key, a database having records indicating a plurality of user identifiers registered to the user identity.
22. The method of claim 1 , wherein the machine identifier comprises at least one of: a media access control (MAC) address or an Internet Protocol (IP) address.
23. The method of claim 1 , wherein the user identifier comprises at least one of: a user login identifier (ID), a username, or an electronic mail address.
24. The method of claim 1 , wherein identifying the plurality of entities in the events comprises:
parsing the events based on a specified data format that specifies which data represent entities in the events.
25. The method of claim 1 , wherein identifying the plurality of entities in the events comprises:
detecting a data format of events.
26. The method of claim 1 , wherein identifying the plurality of entities in the events comprises:
detecting a data format of the events by:
comparing the data format of the events to a list of known event data formats; and
determining a highest probability data format based on a result of said comparing.
27. A computer system comprising:
a communication device; and
a processor configured to:
identify a plurality of entities involved in a plurality of events on a computer network, the plurality of entities including a particular user represented by a user identifier and a machine represented by a machine identifier;
determine that the machine identifier is to be associated with the particular user, based on a statistical analysis of events, of the plurality of events, that satisfy a criterion; and
in response to a determination that the machine identifier is to be associated with the particular user, annotate raw, unstructured machine data of a particular event to include an indication that the particular event is associated with the particular user.
28. The computer system of claim 27 , wherein the processor is further configured to:
update a probability of association between the particular user and the machine identifier upon receiving a new event having at least one of: the machine identifier or the user identifier.
29. A non-transitory machine-readable storage medium for use in a processing system, the non-transitory machine-readable storage medium storing instructions, an execution of which in the processing system causes the processing system to perform operations comprising:
identifying a plurality of entities involved in a plurality of events on a computer network, the plurality of entities including a particular user represented by a user identifier and a machine represented by a machine identifier;
determining that the machine identifier is to be associated with the particular user, based on a statistical analysis of events, of the plurality of events, that satisfy a criterion; and
in response to a determination that the machine identifier is to be associated with the particular user, annotating raw, unstructured machine data of a particular event to include an indication that the particular event is associated with the particular user.
30. The medium of claim 29 , the operations further comprising:
updating a probability of association between the particular user and the machine identifier upon receiving a new event having at least one of: the machine identifier or the user identifier.