Clustering analysis for deduplication of training set samples for machine learning based computer threat analysis
A method, a system, and a computer program product for performing analysis of data to detect presence of malicious code are disclosed. Reduced dimensionality vectors are generated from a plurality of original dimensionality vectors representing features in a plurality of samples. The reduced dimensionality vectors have a lower dimensionality than an original dimensionality of the plurality of original dimensionality vectors. A first plurality of clusters is determined by applying a first clustering algorithm to the reduced dimensionality vectors. A second plurality of clusters is determined by applying a second clustering algorithm to one or more clusters in the first plurality of clusters using the original dimensionality. An exemplar for a cluster in the second plurality of clusters is added to a training set, which is used to train a machine learning model for identifying a file containing malicious code.
1. A computer-implemented method comprising
generating reduced dimensionality vectors from a plurality of original dimensionality vectors representing features in a plurality of samples, the reduced dimensionality vectors having lower dimensionality than an original dimensionality of the plurality of original dimensionality vectors;
first determining a first plurality of clusters, the first determining comprising first applying a first clustering algorithm to the reduced dimensionality vectors;
second determining a second plurality of clusters and a plurality of outliers not forming part of one of the second plurality of clusters, the second determining comprising second applying a second clustering algorithm to original dimensionality vectors corresponding to the reduced dimensionality vectors in the first plurality of clusters, the second plurality of clusters being smaller than the first plurality of clusters;
randomly selecting at least one exemplar from each of at least a portion of the clusters of the second plurality of clusters;
adding the randomly selected exemplars to a training set;
adding the plurality of outliers to the training set; and
training a machine learning model for identifying a file containing malicious code, the training comprising use of the training set;
wherein at least one of the generating, the first determining, the second determining, the adding, and the training is performed by at least one processor of at least one computing system.
2. The method according to claim 1 , wherein the first and second clustering algorithms are same.
3. The method according to claim 1 , where the generating of the reduced dimensionality vectors comprises applying a random projection to the original dimensionality vectors.
4. The method according to claim 3 , wherein the random projection approximately preserves all pairwise distances between the original dimensionality vectors.
5. The method according to claim 3 , wherein the random projection has a predetermined size.
6. The method according to claim 1 , wherein the cluster of the second plurality of clusters has a predetermined radius, wherein pairwise distances between points contained in the cluster of the second plurality of clusters are less than the predetermined radius.
7. The method according to claim 1 , wherein the cluster of the second plurality of clusters has a predetermined radius, wherein pairwise distances between some points contained in the cluster of the second plurality of clusters are greater than the predetermined radius.
8. The method according to claim 1 , wherein the cluster of the second plurality of clusters has a predetermined minimum number of points.
9. A system comprising:
at least one programmable processor; and
memory storing instructions which, when executed by the at least one programmable processor, execute operations comprising:
generating reduced dimensionality vectors from a plurality of original dimensionality vectors representing features in a plurality of samples, the reduced dimensionality vectors having lower dimensionality than an original dimensionality of the plurality of original dimensionality vectors;
first determining a first plurality of clusters, the first determining comprising first applying a first clustering algorithm to the reduced dimensionality vectors;
second determining a second plurality of clusters and a plurality of outliers not forming part of one of the second plurality of clusters, the second determining comprising second applying a second clustering algorithm to original dimensionality vectors corresponding to the reduced dimensionality vectors in the first plurality of clusters, the second plurality of clusters being smaller than the first plurality of clusters;
selecting at least one exemplar from each of at least a portion of the clusters of the second plurality of clusters which correspond to a point approximately close to a center of the associated cluster in the second plurality of clusters;
adding the selected exemplars to a training set;
adding the plurality of outliers to the training set; and
training a machine learning model for identifying a file containing malicious code, the training comprising use of the training set.
10. The system according to claim 9 , wherein the first and second clustering algorithms are same.
11. The system according to claim 9 , where the generating of the reduced dimensionality vectors comprises applying a random projection to the original dimensionality vectors.
12. The system according to claim 11 , wherein the random projection approximately preserves all pairwise distances between the original dimensionality vectors.
13. The system according to claim 11 , wherein the random projection has a predetermined size.
14. A computer program product comprising a non-transitory machine readable medium storing instructions that, when executed by one or more programmable processors, cause the one or more programmable processors to perform operations comprising:
generating reduced dimensionality vectors from a plurality of original dimensionality vectors representing features in a plurality of samples, the reduced dimensionality vectors having lower dimensionality than an original dimensionality of the plurality of original dimensionality vectors;
first determining a first plurality of clusters, the first determining comprising first applying a first clustering algorithm to the reduced dimensionality vectors;
second determining a second plurality of clusters, the second determining comprising second applying a second clustering algorithm to original dimensionality vectors corresponding to the reduced dimensionality vectors in the first plurality of clusters, the second plurality of clusters being smaller than the first plurality of clusters;
adding, from each cluster in the second plurality of clusters, a single exemplar to a training set which is based on a date of creation of the corresponding cluster ; and
training a machine learning model for identifying a file containing malicious code, the training comprising use of the training set.