Cluster detection and elimination in security environments
Disclosed herein are methods, systems, and processes to detect valid clusters and eliminate spurious clusters in cybersecurity-based computing environments. A cluster detection and elimination model is trained by accessing a dataset with raw data that includes data points associated with computing devices in a network and applying two or more different clustering methodologies independently to the dataset. The resulting cluster detection and elimination model is used to compare two or more clusters to determine whether a cluster from one clustering methodology matches another cluster from another clustering methodology based on centroid locations and shared data points.
1. A computer-implemented method, comprising:
accessing a dataset comprising attacker methodologies, credential events, and exploitation events, wherein
the dataset comprises a plurality of data points, and
each data point of the plurality of data points is associated with a computing device of a plurality of computing devices in a network;
applying at least two clustering methodologies independently to the dataset without indicating a value for an initial cluster starting point and assigning an arbitrary value for the initial cluster starting point;
comparing a result of the applying, wherein
the result comprises at least two or more clusters, each derived from separate clustering methodologies of the at least two clustering methodologies,
the comparing comprises determining whether a cluster of the two or more clusters matches another cluster of the two or more clusters based on centroid locations and shared data points regardless of an output label of the cluster and another output label of the another cluster generated by the at least two clustering methodologies, and
the matching indicates that the cluster and the another cluster share an insignificant number of computing devices of the plurality of computing devices based on a first threshold associated with the centroid locations and a second threshold associated with the shared data points; and
based on the comparing,
isolating the cluster and the another cluster as spurious clusters, eliminating the spurious clusters from the result, and
inhibiting the insignificant number of computing devices from being included in a security operation intended for the plurality of computing devices in the network.
2. The computer-implemented method of claim 1 , wherein the value for the initial cluster starting point is unknown or randomized prior to the application of the at least two clustering methodologies independently to the dataset to enable assignment of the arbitrary value for the initial cluster starting point without negatively affecting cluster detection.
3. The computer-implemented method of claim 1 , wherein
the comparing based on the centroid locations comprises determining whether a centroid of the cluster is located near another centroid of the another cluster in feature space relative to other centroids of other clusters of the at least two or more clusters.
4. The computer-implemented method of claim 1 , wherein
the comparing based on the shared data points comprises determining whether the cluster and the another cluster share a significant number of data points of the plurality of data points associated with the plurality of computing devices relative to other clusters of the at least two or more clusters.
5. The computer-implemented method of claim 1 , further comprising:
training a cluster detection and elimination model, wherein
the cluster detection and elimination model is configured to identify and exclude the insignificant number of computing devices from the security operation intended for the plurality of computing devices.
6. The computer-implemented method of claim 1 , wherein the dataset comprises raw security data.
7. A non-transitory computer readable storage medium comprising program instructions executable to:
access a dataset comprising attacker methodologies, credential events, and exploitation events, wherein
the dataset comprises a plurality of data points, and
each data point of the plurality of data points is associated with a computing device of a plurality of computing devices in a network;
apply at least two clustering methodologies independently to the dataset without indicating a value for an initial cluster starting point and assigning an arbitrary value for the initial cluster starting point;
compare a result of the applying, wherein
the result comprises at least two or more clusters, each derived from separate clustering methodologies of the at least two clustering methodologies,
the comparing comprises determining whether a cluster of the two or more clusters matches another cluster of the two or more clusters based on centroid locations and shared data points regardless of an output label of the cluster and another output label of the another cluster generated by the at least two clustering methodologies, and
the matching indicates that the cluster and the another cluster share an insignificant number of computing devices of the plurality of computing devices based on a first threshold associated with the centroid locations and a second threshold associated with the shared data points; and
based on the comparing,
isolate the cluster and the another cluster as spurious clusters, eliminate the spurious clusters from the result, and
inhibit the insignificant number of computing devices from being included in a security operation intended for the plurality of computing devices in the network.
8. The non-transitory computer readable storage medium of claim 7 , wherein
the comparing based on the centroid locations comprises determining whether a centroid of the cluster is located near another centroid of the another cluster in feature space relative to other centroids of other clusters of the at least two or more clusters, and
the comparing based on the shared data points comprises determining whether the cluster and the another cluster share a significant number of data points of the plurality of data points associated with the plurality of computing devices relative to other clusters of the at least two or more clusters.
9. The non-transitory computer readable storage medium of claim 8 , wherein
the value for the initial cluster starting point is unknown or randomized prior to the application of the at least two clustering methodologies independently to the dataset to enable assignment of the arbitrary value for the initial cluster starting point without negatively affecting cluster detection.
10. The non-transitory computer readable storage medium of claim 7 , further comprising:
training a cluster detection and elimination model, wherein
the cluster detection and elimination model is configured to identify and exclude the insignificant number of computing devices from the security operation intended for the plurality of computing devices.
11. A system comprising:
one or more processors; and
a memory coupled to the one or more processors, wherein the memory stores program instructions executable by the one or more processors to:
access a dataset comprising attacker methodologies, credential events, and exploitation events, wherein
the dataset comprises a plurality of data points, and
each data point of the plurality of data points is associated with a computing device of a plurality of computing devices in a network;
apply at least two clustering methodologies independently to the dataset without indicating a value for an initial cluster starting point and assigning an arbitrary value for the initial cluster starting point;
compare a result of the applying, wherein
the result comprises at least two or more clusters, each derived from separate clustering methodologies of the at least two clustering methodologies,
the comparing comprises determining whether a cluster of the two or more clusters matches another cluster of the two or more clusters based on centroid locations and shared data points regardless of an output label of the cluster and another output label of the another cluster generated by the at least two clustering methodologies, and
the matching indicates that the cluster and the another cluster share an insignificant number of computing devices of the plurality of computing devices based on a first threshold associated with the centroid locations and a second threshold associated with the shared data points; and
based on the comparing,
isolate the cluster and the another cluster as spurious clusters, eliminate the spurious clusters from the result, and
inhibit the insignificant number of computing devices from being included in a security operation intended for the plurality of computing devices in the network.
12. The system of claim 11 , wherein
the comparing based on the centroid locations comprises determining whether a centroid of the cluster is located near another centroid of the another cluster in feature space relative to other centroids of other clusters of the at least two or more clusters, and
the comparing based on the shared data points comprises determining whether the cluster and the another cluster share a significant number of data points of the plurality of data points associated with the plurality of computing devices relative to other clusters of the at least two or more clusters.
13. The system of claim 12 , wherein
the value for the initial cluster starting point is unknown or randomized prior to the application of the at least two clustering methodologies independently to the dataset to enable assignment of the arbitrary value for the initial cluster starting point without negatively affecting cluster detection.
14. The system of claim 11 , further comprising:
training a cluster detection and elimination model, wherein
the cluster detection and elimination model is configured to identify and exclude the insignificant number of computing devices from the security operation intended for the plurality of computing devices.