Detection of malware using feature hashing
Data is analyzed using feature hashing to detect malware. A plurality of features in a feature set is hashed. The feature set is generated from a sample. The sample includes at least a portion of a file. Based on the hashing, one or more hashed features are indexed to generate an index vector. Each hashed feature corresponds to an index in the index vector. Using the index vector, a training dataset is generated. Using the training dataset, a machine learning model for identifying at least one file having a malicious code is trained.
1. A computer-implemented method comprising:
hashing a plurality of features in a feature set, wherein the feature set is generated from a sample and the sample includes at least a portion of a file;
indexing, based on the hashing, one or more hashed features to generate a plurality of index vectors each corresponding to a different feature set, wherein each hashed feature corresponds to an index in the index vector;
generating, using the index vector, a training dataset; and
training, using the training dataset, a machine learning model for identifying at least one file having a malicious code;
wherein:
the indexing further comprises determining a plurality of most frequently occurring indexes in the plurality of index vectors; and
the machine learning model is trained using the training dataset generated based on the plurality of most frequently occurring indexes.
2. The method according to claim 1 , wherein a format of the file is selected from a group consisting of: a portable executable format, a document format, a file format, an executable format, a script format, an image format, a video format, and an audio format.
3. The method according to claim 1 , wherein the index includes a value corresponding to a hashed feature and a sign attribute.
4. The method according to claim 3 , wherein the value is determined based on a name of each hashed feature.
5. The method according to claim 4 , wherein the sign attribute includes at least one of the following: a positive value and a negative value.
6. The method according to claim 5 , further comprising
determining, based on the hashing, a first feature, having a first index, colliding with a second feature, having a second index;
assigning a first sign attribute to the first index and a second sign attribute to the second index, the first sign attribute is different from the second sign attribute; and
generating, using assigned sign attributes, the index vector having the first index and the second index.
7. The method according to claim 5 , wherein a combination of each index and a sign attribute for each feature in the plurality of features corresponds to a predetermined position in the index vector.
8. The method according to claim 1 , wherein the index vector has a predetermined size.
9. The method according to claim 1 , wherein at least one of the hashing, the indexing, the generating, and the training is performed by at least one processor of at least one computing system, wherein the computing system comprises: at least one software component, at least one hardware component, and any combination thereof.
10. A system comprising:
at least one programmable data processor;
memory storing instructions which, when executed by the at least one programmable data processor, result in operations comprising:
hashing a plurality of features in a feature set, wherein the feature set is generated from a sample and the sample includes at least a portion of a file;
indexing, based on the hashing, one or more hashed features to generate a plurality of index vectors each corresponding to a different feature set, wherein each hashed feature corresponds to an index in the index vector;
generating, using the index vector, a training dataset; and
training, using the training dataset, a machine learning model for identifying at least one file having a malicious code;
wherein:
the indexing further comprises determining a plurality of most frequently occurring indexes in the plurality of index vectors; and
the machine learning model is trained using the training dataset generated based on the plurality of most frequently occurring indexes.
11. The system according to claim 10 , wherein a format of the file is selected from a group consisting of: a portable executable format, a document format, a file format, an executable format, a script format, an image format, a video format, and an audio format.
12. The system according to claim 10 , wherein the index includes a value corresponding to a hashed feature and a sign attribute.
13. The system according to claim 12 , wherein the value is determined based on a name of each hashed feature.
14. The system according to claim 13 , wherein the sign attribute includes at least one of the following: a positive value and a negative value.
15. The system according to claim 14 , wherein the operations further comprise:
determining, based on the hashing, a first feature, having a first index, colliding with a second feature, having a second index;
assigning a first sign attribute to the first index and a second sign attribute to the second index, the first sign attribute is different from the second sign attribute; and
generating, using assigned sign attributes, the index vector having the first index and the second index;
wherein a combination of each index and a sign attribute for each feature in the plurality of features corresponds to a predetermined position in the index vector.
16. A non-transitory computer program product storing instructions which, when executed by comprising:
at least one programmable data processor;
memory storing instructions which, when executed by the at least one programmable data processor, result in operations comprising:
hashing a plurality of features in a feature set, wherein the feature set is generated from a sample and the sample includes at least a portion of a file;
indexing, based on the hashing, one or more hashed features to generate a plurality of index vectors each corresponding to a different feature set, wherein each hashed feature corresponds to an index in the index vector;
generating, using the index vector, a training dataset; and
training, using the training dataset, a machine learning model for identifying at least one file having a malicious code;
wherein:
the indexing further comprises determining a plurality of most frequently occurring indexes in the plurality of index vectors; and
the machine learning model is trained using the training dataset generated based on the plurality of most frequently occurring indexes.
17. The non-transitory computer program product according to claim 16 , wherein a format of the file comprises at least one of a portable executable format, a document format, a file format, an executable format, a script format, an image format, a video format, and an audio format.
18. The non-transitory computer program product according to claim 16 , wherein the index includes a value corresponding to a hashed feature and a sign attribute.
19. The non-transitory computer program product according to claim 18 , wherein the value is determined based on a name of each hashed feature.
20. The non-transitory computer program product according to claim 19 , wherein the sign attribute includes at least one of the following: a positive value and a negative value.