Systems and methods for determining measurements of similarity between various types of data
Systems and methods are provided to measure the similarity between a first and second data sample. The method can include creating a plurality of k-mers from the first data sample, each k-mer having a first length; generating a first vector from the plurality of k-mers by processing the plurality of k-mers with a plurality of hash functions; calculating a similarity level between the first and second data sample by comparing the first vector to a second vector, the second vector representing the second data sample; and based on the similarity level, determining a maliciousness level of the first data sample.
1. A computer-implemented method for classifying a data sample comprising:
creating a plurality of k-mers from the data sample, each k-mer having a first length;
generating a vector from the plurality of k-mers by processing the plurality of k-mers with a plurality of hash functions;
comparing the vector to a plurality of vector clusters, each vector cluster of the plurality of vector clusters comprising one or more vectors representing at least one other data sample that share common characteristics;
identifying membership of the data sample to one of the plurality of vector clusters by determining a plurality of distance metrics between the vector and each vector cluster of the plurality of vector clusters; and
based on the plurality of distance metrics, determining a characteristic of the data sample.
2. The computer-implemented method of claim 1 , wherein the at least one other sample is associated with malware or a program that causes damage.
3. The computer-implemented method of claim 1 , wherein identifying membership of the data sample to one of the plurality of vector clusters comprises:
iterating through the plurality of clusters based on the plurality of distance metrics.
4. The computer-implemented method of claim 1 , wherein determining the at least one distance metric comprises approximating a similarity between the vector and the at least one other vector.
5. The computer-implemented method of claim 4 , wherein approximating the similarity comprises using a MinHash function on each of the plurality of hash functions.
6. The computer-implemented method of claim 1 comprising, prior to generating the vector, at least one of manipulating, modifying, or selecting the plurality of k-mers.
7. The computer-implemented method of claim 6 , where manipulating the plurality of k-mers comprises at least one of removing at least one k-mer or mapping two or more k-mers to a representative value.
8. The computer-implemented method of claim 1 comprising, prior to creating the plurality of k-mers from the data sample, measuring a level of information content in the data sample.
9. The computer-implemented method of claim 8 comprising, in response to determining that the level of information content is above a pre-defined threshold, creating the plurality of k-mers from the data sample.
10. The computer-implemented method of claim 8 , wherein measuring the level of information content comprises at least one of calculating an entropy of the data sample or calculating a cardinality of the data sample.
11. A computer-implemented method for preventing data leaks from a protected environment, the method comprising:
detecting, at an egress gateway, a data sample being transmitted outside of the protected environment, the data sample comprising confidential information;
creating a plurality of k-mers from the data sample, each k-mer having a first length;
generating a vector from the plurality of k-mers by processing the plurality of k-mers with a plurality of hash functions;
comparing the vector to a set of protected vectors to determine at least one distance metric, the set of protected vectors representing a plurality of data samples that are to remain within the protected environment; and
based on the at least one distance metric, preventing the data sample from leaving the protected environment.
12. The computer-implemented method of claim 11 , wherein determining the at least one distance metric comprises approximating a similarity between the vector and the set of protected vectors.
13. The computer-implemented method of claim 12 , wherein approximating the similarity comprises using a MinHash function on each of the plurality of hash functions.
14. The computer-implemented method of claim 11 , where the first length is at least three bytes.
15. The computer-implemented method of claim 11 comprising determining the first length based on at least one statistical property of the data sample.
16. The computer-implemented method of claim 11 comprising, prior to generating the vector, at least one of manipulating, modifying, or selecting the plurality of k-mers.
17. The computer-implemented method of claim 16 , where manipulating the plurality of k-mers comprises at least one of removing at least one k-mer or mapping two or more k-mers to a representative value.
18. The computer-implemented method of claim 11 , wherein the plurality of k-mers is generated such that at least one of the plurality of k-mers partially overlaps another.
19. A computer-implemented method for vectorizing a data sample comprising:
executing the data sample;
collecting runtime data associated with the execution of the data sample;
creating a plurality of k-mers from the runtime data, each k-mer having a first length;
mapping the plurality of k-mers to a plurality of integers with a plurality of hash functions;
generating a vector from the mapped plurality of k-mers with a MinHash function;
comparing the vector to a plurality of vector clusters, each vector cluster of the plurality of vector clusters comprising one or more vectors representing at least one other data sample that share common characteristics;
identifying membership of the data sample to one of the plurality of vector clusters by determining a plurality of distance metrics between the vector and each vector cluster of the plurality of vector clusters; and
based on the plurality of distance metrics, determining a characteristic of the data sample.
20. The computer-implemented method of claim 19 , wherein the data sample is executed in a sandbox.
21. The computer-implemented method of claim 19 , wherein the runtime data comprises at least one of memory buffers, intermediate files, or API tracing information.
22. The computer-implemented method of claim 19 , wherein determining the plurality of distance metrics comprises approximating a similarity between the vector and each vector cluster of the plurality of vector clusters.
23. The computer-implemented method of claim 22 , wherein approximating the similarity comprises using the MinHash function on each of the plurality of hash functions.
24. The computer-implemented method of claim 19 , where the first length is at least three bytes.
25. The computer-implemented method of claim 19 comprising determining the first length based on at least one statistical property of the data sample.
26. The computer-implemented method of claim 19 comprising, prior to generating the vector, at least one of manipulating, modifying, or selecting the plurality of k-mers.
27. The computer-implemented method of claim 26 , where manipulating the plurality of k-mers comprises at least one of removing at least one k-mer or mapping two or more k-mers to a representative value.
28. The computer-implemented method of claim 26 , wherein the plurality of k-mers is generated such that at least one of the plurality of k-mers partially overlaps another.