Machine-learning based approach for malware sample clustering
Systems and methods for a machine learning based approach for identification of malware using static analysis and a machine-learning based automatic clustering of malware are provided. According to various embodiments of the present disclosure, a processing resource of a computer system receives a potential malware sample. A plurality of feature vectors is extracted from the potential malware sample and is converted into an input vector. A byte sequence is generated by walking a plurality of decision trees based on the input vector. Further, a hash value for the byte sequence is calculated and a determination is made regarding whether the hash value matches a malware hash value of a plurality of malware hash values corresponding to a known malware sample. Upon said determination being affirmative, the potential malware sample is classified as malware and is associated with a malware family of the known malware sample.
1. A method comprising:
receiving, by a processing resource of a computer system, a potential malware sample;
extracting, by the processing resource, a plurality of feature vectors from the potential malware sample, wherein the plurality of feature vectors represent values of static features of the potential malware sample;
converting, by the processing resource, the plurality of feature vectors into an input vector;
generating, by the processing resource, a byte sequence by walking a plurality of decision trees based on the input vector, wherein the plurality of decision trees are associated with a machine-learning model that has been trained based on the static features of a set of known malware samples, wherein the machine-learning model comprises a Random Forest model and wherein the plurality of decision trees comprises binary decision trees;
calculating, by the processing resource, a hash value for the byte sequence;
determining, by the processing resource, whether the hash value matches a malware hash value of a plurality of malware hash values corresponding to a known malware sample of the set of known malware samples; and
when said determining is affirmative, classifying, by the processing resource, the potential malware sample as malware and associating the malware with a malware family of the known malware sample.
2. The method of claim 1 , further comprising when said determining is negative:
determining, by the processing resource, whether the byte sequence meets a predetermined or configurable threshold of similarity with a malware byte sequence of a particular known malware sample of the set of known malware samples; and
when said determining whether the byte sequence meets a predetermined or configurable threshold of similarity is affirmative:
classifying, by the processing resource, the potential malware sample as malware; and
treating, by the processing resource, the malware as a variant of a malware family of the particular known malware sample by adding the potential malware sample to the set of known malware samples as part of a new cluster within the set of known malware samples.
3. The method of claim 1 , wherein said walking the plurality of decision trees based on the input vector comprises:
for each a binary decision tree of the plurality of decision trees:
evaluating an expression involving one or more features of the plurality of features associated with a current node starting with a root node of the binary decision tree and ending at a leaf node of the binary decision tree;
when said evaluating causes a left branch of the current node to be taken, assigning a first value to a portion of the byte sequence corresponding to the current node; and
when said evaluating causes a right branch of the current node to be taken, assigning a second value to the portion of the byte sequence.
4. The method of claim 1 , wherein the binary decision trees are Classification and Regression Trees (CART), where each of a node of the CART trees has at most two branches.
5. The method of claim 1 , wherein when the hash value of the malware matches to at least one of the malware hash value of the plurality of malware hash values corresponding to the at least one of known malware sample of the set of known malware samples, associating, by the processing resource, the malware with the malware family of the matched at least one of known malware sample.
6. The method of claim 1 , wherein the hash value is calculated by concatenating the generated byte sequence to form a unique predefined byte sequence.
7. The method of claim 1 , wherein the plurality of feature vectors comprises any or a combination of entry point information, an import table, resource information, a DOTNET structural data, and a set of text strings pertaining to the potential malware sample.
8. The method of claim 1 , wherein the processing resource is configured on a cloud based service.
9. A non-transitory computer-readable storage medium embodying a set of instructions, which when executed by a processing resource of a computing system, causes the processing resource to perform a method comprising:
receiving a potential malware sample;
extracting a plurality of feature vectors from the potential malware sample, wherein the plurality of feature vectors represent values of static features of the potential malware sample;
converting the plurality of feature vectors into an input vector;
generating a byte sequence by walking a plurality of decision trees based on the input vector, wherein the plurality of decision trees are associated with a machine-learning model that has been trained based on the static features of a set of known malware samples, wherein the machine-learning model comprises a Random Forest model and wherein the plurality of decision trees comprises binary decision trees;
calculating a hash value for the byte sequence;
determining whether the hash value matches a malware hash value of a plurality of malware hash values corresponding to a known malware sample of the set of known malware samples; and
when said determining is affirmative, classifying the potential malware sample as malware and associating the malware with a malware family of the known malware sample.
10. The non-transitory computer-readable storage medium of claim 9 , further comprising when said determining is negative:
determining whether the byte sequence meets a predetermined or configurable threshold of similarity with a malware byte sequence of a particular known malware sample of the set of known malware samples; and
when said determining whether the byte sequence meets a predetermined or configurable threshold of similarity is affirmative:
classifying the potential malware sample as malware; and
treating the malware as a variant of a malware family of the particular known malware sample by adding the potential malware sample to the set of known malware samples as part of a new cluster within the set of known malware samples.
11. The non-transitory computer-readable storage medium of claim 9 , wherein said walking the plurality of decision trees based on the input vector comprises:
for each a binary decision tree of the plurality of decision trees:
evaluating an expression involving one or more features of the plurality of features associated with a current node starting with a root node of the binary decision tree and ending at a leaf node of the binary decision tree;
when said evaluating causes a left branch of the current node to be taken, assigning a first value to a portion of the byte sequence corresponding to the current node; and
when said evaluating causes a right branch of the current node to be taken, assigning a second value to the portion of the byte sequence.
12. The non-transitory computer-readable storage medium of claim 9 , wherein the binary decision trees are Classification and Regression Trees (CART), where each of a node of the CART trees has at most two branches.
13. The non-transitory computer-readable storage medium of claim 9 , wherein when the hash value of the malware matches to at least one of the malware hash value of the plurality of malware hash values corresponding to the at least one of known malware sample of the set of known malware samples, associating, by the processing resource, the malware with the malware family of the matched at least one of known malware sample.
14. The non-transitory computer-readable storage medium of claim 9 , wherein the hash value is calculated by concatenating the generated byte sequence to a unique predefined byte sequence.
15. The non-transitory computer-readable storage medium of claim 9 , wherein the plurality of feature vectors comprises any or a combination of entry point information, an import table, resource information, a DOTNET structural data, and a set of text strings pertaining to the potential malware sample.
16. The non-transitory computer-readable storage medium of claim 9 , wherein the processing resource is configured on a cloud based service.
17. A computer system comprising:
a processing resource;
a network interface; and
a memory, storing source code for a plurality of modules executable by the processing resource and the plurality of modules comprising:
a first module to receive potential malware sample, extract a plurality of feature vectors from the potential malware sample, wherein the plurality of feature vectors represent values of static features of the potential malware sample;
a second module to convert the plurality of feature vectors into an input vector;
a third module to generate a byte sequence by walking a plurality of decision trees based on the input vector, wherein the plurality of decision trees are associated with a machine-learning model that has been trained based on the static features of a set of known malware samples, wherein the machine-learning model comprises a Random Forest model and wherein the plurality of decision trees comprises binary decision trees;
a fourth module to calculate a hash value for the byte sequence;
a fifth module to determine whether the hash value matches a malware hash value of a plurality of malware hash values corresponding to a known malware sample of the set of known malware samples; and
a sixth module, when said determining is affirmative, to claissify the potential malware sample as malware and associating the malware with a malware family of the known malware sample.