Large scale malware sample identification

1. An apparatus, comprising:

a memory element operable to store instructions; and

a processor operable to execute the instructions, such that the apparatus is configured to:

receive a feature vector of an unknown sample, wherein the feature vector includes Jaccard-compatible properties, wherein at least a portion of the Jaccard-compatible properties include an operating system the unknown sample is designed to operate on, an indicator that indicates if the unknown sample is an executable file, and a type of file for the unknown sample, and non-Jaccard distance-compatible features where at least a portion of the non-Jaccard distance compatible features are numerical values;

compute a MinHash of the unknown sample based on the Jaccard-compatible properties;

query a MinHash Locality Sensitive Hashing (LSH) Forest of known samples with the MinHash of the unknown sample to identify a first subset of known samples that are similar to the unknown sample;

receive, for each individual known sample in the first subset, a feature vector having non-Jaccard distance-compatible features;

compute, for each individual known sample in the first subset, a first sub-distance between the unknown sample and the known sample using a first non-Jaccard distance-compatible feature and a second sub-distance between the unknown sample and the known sample using a second non-Jaccard distance-compatible feature;

calculate, for each individual known sample in the first subset, an overall distance between the unknown sample and the known sample by combining the first and the second sub-distances;

identify, based on the overall distance between the unknown sample and each individual known sample in the first subset, a second subset of known samples that are most similar to the unknown sample; and

classify the unknown sample based on the second subset of known samples.

2. The apparatus of claim 1 , further configured to:

assign a remediation to the classified unknown sample.

3. The apparatus of claim 1 , wherein the unknown sample and the known samples are malware.

4. The apparatus of claim 1 , wherein identifying the second subset of known samples includes selecting the known samples having the overall distance that is equal to or less than a threshold.

5. The apparatus of claim 1 , wherein identifying the second subset of known samples includes selecting a number of known samples with the lowest overall distance.

6. The apparatus of claim 1 , wherein a number of known samples in the second subset is between 10 and 25.

7. The apparatus of claim 1 , wherein a number of known samples in the first subset is between 250 and 500.

8. The apparatus of claim 1 , wherein the first non-Jaccard distance-compatible feature is Euclidean-compatible and the second non-Jaccard distance-compatible feature is Hamming-compatible.

9. The apparatus of claim 1 , further configured to:

receive a unique identifier of the unknown sample;

query an index of the MinHash LSH Forest using the unique identifier to determine whether the unknown sample is in the MinHash LSH Forest; and

add the unknown sample to the MinHash LSH Forest based on a determination that the unknown sample is not in the MinHash LSH Forest.

10. At least one non-transitory computer-readable medium comprising one or more instructions that when executed by a processor, cause the processor to:

receive a feature vector of an unknown malware, wherein the feature vector includes Jaccard-compatible properties, wherein at least a portion of the Jaccard-compatible properties include an operating system the unknown malware is designed to operate on, an indicator that indicates if the unknown malware is an executable file, a type of file for the unknown malware, and/or an indicator that indicates if the unknown malware is packed or unpacked, a first non-Jaccard sub-distance feature, and a second non-Jaccard sub-distance feature;

compute a MinHash of the unknown malware;

identify a first subset of known malware, wherein identifying the first subset includes querying a MinHash Locality Sensitive Hashing (LSH) Forest of known malware with the MinHash of the unknown malware;

receive a feature vector for each individual known malware in the first subset, wherein the feature vector includes a first non-Jaccard sub-distance feature and a second non-Jaccard sub-distance feature;

calculate a first sub-distance and a second sub-distance between the unknown malware and each individual known malware in the first subset, wherein the first sub-distance is based on the first non-Jaccard sub-distance features and the second sub-distance is based on the second non-Jaccard sub-distance features;

calculate a Custom Distance Metric (CDM) for each individual known malware in the first subset by combining the first and the second sub-distances;

identify, using the calculated CDM for each individual known malware in the first subset, a second subset of known malware that are most similar to the unknown malware; and

classify the unknown malware based on the second subset of known malware.

11. The at least one non-transitory computer-readable medium of claim 10 , further comprising one or more instructions that when executed by a processor, cause the processor to:

assign a remediation to the classified unknown malware.

12. The at least one non-transitory computer-readable medium of claim 10 , wherein identifying the second subset of known malware includes selecting the known malware having the CDM that is equal to or less than a threshold.

13. The at least one non-transitory computer-readable medium of claim 10 , wherein identifying the second subset of known malware includes selecting a number of known malware with the lowest CDM.

14. The at least one non-transitory computer-readable medium of claim 10 , wherein classifying the known malware includes assigning a type, a family, an author, or a source location.

15. The at least one non-transitory computer-readable medium of claim 10 , further comprising one or more instructions that when executed by a processor, cause the processor to:

receive a unique identifier for the unknown malware;

query an index of the MinHash LSH Forest using the unique identifier to determine whether the unknown malware is in the MinHash LSH Forest; and

add the unknown malware to the MinHash LSH Forest based on a determination that the unknown malware is not in the MinHash LSH Forest.

16. A method, comprising:

creating a MiniHash of an unknown malware sample based on a feature vector of the unknown malware sample, wherein the feature vector includes Jaccard-compatible properties and at least a portion of the Jaccard-compatible properties include an operating system the unknown malware is designed to operate on, an indicator that indicates if the unknown malware is an executable file, a type of file for the unknown malware, and an indicator that indicates if the unknown malware is packed or unpacked,

receiving a MinHash for an unknown malware sample;

querying a MinHash Locality Sensitive Hashing (LSH) Forest using the MinHash for the unknown malware sample to identify a first set of similar samples, wherein the MinHash LSH Forest is based on a plurality of known malware samples, and wherein each known malware sample in the plurality of known malware samples includes a unique hashing identifier and a MinHash;

receiving a first non-Jaccard distance-compatible feature and a second non-Jaccard distance-compatible feature for the unknown malware sample;

receiving a first non-Jaccard distance-compatible feature and a second non-Jaccard distance-compatible feature for each known malware sample in the first set of similar samples;

calculating a total distance between the unknown malware sample and each known malware sample in the first set of similar samples, wherein the total distance is a combination of a first distance and a second distance, and wherein the first distance is calculated using the first non-Jaccard distance-compatible features and the second distance is calculated using the second non-Jaccard distance-compatible features;

identifying, based on the total distance for each known malware sample, a second set of similar samples; and

classifying the unknown malware sample based on the second set of similar samples.

17. The method of claim 16 , further comprising:

receiving a unique identifier of the classified unknown malware sample;

querying an index of the MinHash LSH Forest using the unique identifier to determine whether the classified unknown malware sample is in the MinHash LSH Forest; and

adding the classified unknown malware sample to the MinHash LSH Forest based on a determination that the classified unknown malware sample is not in the MinHash LSH Forest.

18. The method of claim 16 , further comprising:

assigning a remediation to the classified unknown malware sample.

19. The method of claim 16 , wherein the first non-Jaccard distance-compatible features are Euclidean-type and the second non-Jaccard distance-compatible features are non-Euclidean-type.

20. The method of claim 16 , wherein a number of known samples in the second set is between 10 and 25.