Cluster-based scheduling of security operations
Disclosed herein are methods, systems, and processes to detect valid clusters and eliminate spurious clusters in cybersecurity-based computing environments. A cluster detection and elimination model is trained by accessing a dataset with raw data that includes data points associated with computing devices in a network and applying two or more different clustering methodologies independently to the dataset. The resulting cluster detection and elimination model is used to compare two or more clusters to determine whether a cluster from one clustering methodology matches another cluster from another clustering methodology based on centroid locations and shared data points.
1. A computer-implemented method, comprising:
accessing a dataset of a plurality of data points, wherein the data points represent features of a plurality of computing devices in a network;
applying at least two clustering methodologies independently to the dataset to generate at least two cluster sets;
identifying two clusters for comparison analysis, including a first cluster in one of the at least two cluster sets generated by a first clustering methodology and a second cluster in another of the at least two cluster sets generated by a second clustering methodology;
comparing the two clusters to determine whether the two clusters sufficiently match, wherein the determination is made based on an adjustable threshold of one or more match indicators;
determining whether at least one of the two clusters is a valid cluster based on whether the two clusters sufficiently match; and
scheduling a security operation on computing devices in valid clusters in the network, wherein computing devices in invalid clusters are excluded from the security operation to save computing resources used by the security operation.
2. The computer-implemented method of claim 1 ,
wherein
the applying of the at least two clustering methodologies is performed without receiving a value for an initial cluster starting point, and
the method comprises determining a random value as the initial cluster starting point prior to the application of the at least two clustering methodologies.
3. The computer-implemented method of claim 1 , wherein at least one of the match indicators is determined based on respective centroid locations of the two clusters.
4. The computer-implemented method of claim 1 , wherein at least one of the match indicators is determined based on a number of shared data points between the two clusters.
5. The computer-implemented method of claim 1 , wherein the features of
a computing device in the dataset include one or more of:
a number of connections to the computing device,
a number of connection attempts to the computing device,
a number of other assets associated with the computing device,
a number of users associated with the computing device,
a number of logins made to the computing device, and
a number of scans of the computing device.
6. The computer-implemented method of claim 1 , wherein the at least two clustering methodologies comprise one or more of:
an Agglomerative algorithm,
a K-Means algorithm,
a Mean-Shift algorithm,
a Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm, and
an Expectation Maximization (EM) algorithm.
7. The computer-implemented method of claim 1 , further comprising:
keeping the valid clusters for performance of additional security operations; and
eliminating the invalid clusters so that they are excluded from additional security operations.
8. The computer-implemented method of claim 7 , wherein keeping the valid clusters comprises labeling the valid clusters as valid in a storage system.
9. The computer-implemented method of claim 1 , wherein
the network is a private network of an organization, and
the method further comprises collecting the dataset from the network over a public network.
10. The computer-implemented method of claim 1 , further comprising: using the dataset to detect abnormal actions in the network or vulnerabilities in the network.
11. A system comprising:
one or more computing devices configured to:
access a dataset of a plurality of data points, wherein the data points represent features of a plurality of computing devices in a network;
apply at least two clustering methodologies independently to the dataset to generate at least two cluster sets;
identify two clusters for comparison analysis, including a first cluster in one of the at least two cluster sets generated by a first clustering methodology and a second cluster in another of the at least two cluster sets generated by a second clustering methodology;
compare the two clusters to determine whether the two clusters sufficiently match, wherein the determination is made based on an adjustable threshold of one or more match indicators;
determine whether at least one of the two clusters is a valid cluster based on whether the two clusters sufficiently match; and
schedule a security operation on computing devices in valid clusters in the network, wherein computing devices in invalid clusters are excluded from the security operation to save computing resources used by the security operation.
12. The system of claim 11 , wherein
the at least two clustering methodologies are applied without receiving a value for an initial cluster starting point, and
the one or more computing devices are configured to determine a random value as the initial cluster starting point prior to the application of the at least two clustering methodologies.
13. The system of claim 11 , wherein at least one of the match indicators is determined based on respective centroid locations of the two clusters.
14. The system of claim 11 , wherein at least one of the match indicators is determined based on a number of shared data points between the two clusters.
15. The system of claim 11 , wherein the one or more computing devices is configured to:
keep the valid clusters for performance of additional security operations; and
eliminate the invalid clusters so that they are excluded from additional security operations.
16. The system of claim 15 , wherein the one or more computing devices is configured to keep the valid clusters by labeling the valid clusters as valid in a storage system.
17. The system of claim 11 , wherein
the network is a private network of an organization, and
the one or more computing devices are configured to collect the dataset from the network over a public network.
18. The system of claim 11 , wherein the one or more computing devices are configured to using the dataset to detect abnormal actions in the network or vulnerabilities in the network.
19. One or more non-transitory computer readable media storing program instructions that when executed on one or more processors cause the one or more processors to:
access a dataset of a plurality of data points, wherein the data points represent features of a plurality of computing devices in a network;
apply at least two clustering methodologies independently to the dataset to generate at least two cluster sets;
identify two clusters for comparison analysis, including a first cluster in one of the at least two cluster sets generated by a first clustering methodology and a second cluster in another of the at least two cluster sets generated by a second clustering methodology;
compare the two clusters to determine whether the two clusters sufficiently match, wherein the determination is made based on an adjustable threshold of one or more match indicators;
determine whether at least one of the two clusters is a valid cluster based on whether the two clusters sufficiently match; and
schedule a security operation on computing devices in valid clusters in the network, wherein computing devices in invalid clusters are excluded from the security operation to save computing resources used by the security operation.
20. The one or more non-transitory computer readable media of claim 19 , wherein the program instructions when executed on the one or more processors cause the one or more processors to:
keep the valid clusters for performance of additional security operations; and
eliminate the invalid clusters so that they are excluded from additional security operations.