Detection of user behavior deviation from defined user groups
A machine learning-based technique for user behavior analysis that detects when users deviate from expected behavior. In this approach, a set of user groups are provided, preferably based on information provided from a user registry. A set of training data for each of the set of user groups is then obtained, preferably by collecting security events generated for a collection of the users over a given time period (e.g., a last thirty (30) days). A machine learning system is then trained using the set of training data to produce a model that includes a set of clusters in user behavior model, wherein a cluster is a learned user group that corresponds to a defined user group. Once the model is built, it is used to identify users that deviate from their expected group behavior. In particular, the system compares a current behavior of a user against the model and flags anomalous behavior. The user behavior analysis may be implemented in a security platform, such as a SIEM.
1. A method for detecting user behavior deviation in an enterprise network, comprising:
defining a set of user groups based on user attribute information received from a user directory of the enterprise;
receiving from an application security events for a collection of users whose user attribute information is used to define the set of user groups, and generating a set of training data;
training a machine learning system using the set of training data set to generate a set of clusters in a model of user behavior by: (i) transforming the set of training data to generate transformed training data, (ii) applying a clustering model to the transformed training data to derive a set of learned groupings, and (iii) computing measures of distance between the set of learned groupings and the set of user groups, wherein a cluster is a learned user group that corresponds to a defined user group in the set of user groups; and
comparing a behavior of a user against the model to detect a deviation from an expected behavior for the user as represented by one of the clusters.
2. The method as described in claim 1 wherein the application is a security information and event monitoring (SIEM) application.
3. The method as described in claim 2 wherein the security events comprise low-level security events ingested from the SIEM.
4. The method as described in claim 1 wherein
the set of training data is transformed according to a latent allocation model;
and
wherein the measures of distance are computed to generate a probability distribution function.
5. The method as described in claim 4 wherein the clustering model is a Gaussian mixture model and comparing the behavior of the user includes:
receiving new data representing security events associated with a set of users that include the user;
re-applying the latent allocation model and the Gaussian mixture model to the new data;
for the user, computing a measure of distance between a learned grouping that includes the user and the user group associated with the user; and
determining whether the measure of distance computed for the user exceeds a given probability threshold.
6. The method as described in claim 5 wherein the measure of distance computed for the new user is a Jaccard distance.
7. An apparatus, comprising:
a processor;
computer memory holding computer program instructions executed by the processor to detect user behavior deviation in an enterprise network, the computer program instructions configured to:
define a set of user groups based on user attribute information received from a user directory of the enterprise;
receive from an application security events for a collection of users whose user attribute information is used to define the set of user groups, and generate a set of training data;
train a machine learning system using the set of training data set to generate a set of clusters in a model of user behavior by: (i) transforming the set of training data to generate transformed training data, (ii) applying a clustering model to the transformed training data to derive a set of learned groupings, and (iii) computing measures of distance between the set of learned groupings and the set of user groups, wherein a cluster is a learned user group that corresponds to a defined user group in the set of user groups; and
compare a behavior of a user against the model to detect a deviation from an expected behavior for the user as represented by one of the clusters.
8. The apparatus as described in claim 7 wherein the application is a security information and event monitoring (SIEM) application.
9. The apparatus as described in claim 8 wherein the security events comprise low-level security events ingested from the SIEM.
10. The apparatus as described in claim 7 wherein
the set of training data is transformed according to a latent allocation model;
and
wherein the measures of distance are computed to generate a probability distribution function.
11. The apparatus as described in claim 10 wherein the clustering model is a Gaussian mixture model and the computer program instructions to compare the behavior of the user includes computer program instructions further configured to:
receive new data representing security events associated with a set of users that include the user;
re-apply the latent allocation model and the Gaussian mixture model to the new data;
for the user, compute a measure of distance between a learned grouping that includes the user and the user group associated with the user; and
determine whether the measure of distance computed for the user exceeds a given probability threshold.
12. The apparatus as described in claim 11 wherein the measure of distance computed for the new user is a Jaccard distance.
13. A computer program product in a non-transitory computer readable medium for use in a data processing system to detect user behavior deviation in an enterprise network, the computer program product holding computer program instructions that, when executed by the data processing system, are configured to:
define a set of user groups based on user attribute information received from a user directory of the enterprise;
receive from an application security events for a collection of users whose user attribute information is used to define the set of user groups, and generate a set of training data;
train a machine learning system using the set of training data set to generate a set of clusters in a model of user behavior by: (i) transforming the set of training data to generate transformed training data, (ii) applying a clustering model to the transformed training data to derive a set of learned groupings, and (iii) computing measures of distance between the set of learned groupings and the set of user groups, wherein a cluster is a learned user group that corresponds to a defined user group in the set of user groups; and
compare a behavior of a user against the model to detect a deviation from an expected behavior for the user as represented by one of the clusters.
14. The computer program product as described in claim 13 wherein the application is a security information and event monitoring (SIEM) application.
15. The computer program product as described in claim 14 wherein the security events comprise low-level security events ingested from the SIEM.
16. The computer program product as described in claim 13 wherein
the set of training data is transformed according to a latent allocation model;
and
wherein the measures of distance are computed to generate a probability distribution function.
17. The computer program product as described in claim 16 wherein the clustering model is a Gaussian mixture model and the computer program instructions to compare the behavior of the user includes computer program instructions further configured to:
receive new data representing security events associated with a set of users that include the user;
re-apply the latent allocation model and the Gaussian mixture model to the new data;
for the user, compute a measure of distance between a learned grouping that includes the user and the user group associated with the user; and
determine whether the measure of distance computed for the user exceeds a given probability threshold.
18. The computer program product as described in claim 17 wherein the measure of distance computed for the new user is a Jaccard distance.