System and method for automated machine-learning, zero-day malware detection
Improved systems and methods for automated machine-learning, zero-day malware detection. Embodiments include a system and method for detecting malware using multi-stage file-typing and, optionally pre-processing, with fall-through options. The system and method receive a set of training files which are each known to be either malign or benign, partition the set of training files into a plurality of categories based on file-type, in which the partitioning file-types a subset of the training files into supported file-type categories, train file-type specific classifiers that distinguish between malign and benign files for the supported file-type categories of files, associate supported file-types with a file-type processing chain that includes a plurality of file-type specific classifiers corresponding to the supported file-types, train a generic file-type classifier that applies to file-types that are not supported file-types, and construct a composite classifier using the file-type specific classifiers and the generic file-type classifier.
1. A method comprising:
determining at least a first file-type and a second file-type associated with a plurality of files;
determining, via a first classifier of a plurality of classifiers that is trained to determine whether files of the first file-type are at least malign or benign, a first classification for a first file of the plurality of files, wherein the first file is of the first file-type; and
determining, via a second classifier of the plurality of classifiers that is trained to determine whether files of a plurality of file-types are at least malign or benign and based on a second file of the plurality of files being of the second file-type that is different than the first file-type, a second classification for the second file.
2. The method of claim 1 , wherein the determining at least the first file-type further comprises:
determining, based on at least one byte sequence at an offset in the first file, the first file-type; and
determining, based on the first file-type, whether an internal structure of the first file matches an expected internal structure.
3. The method of claim 1 , wherein training each classifier of the plurality of classifiers comprises:
determining, based on at least one feature from each training file of a plurality of training files, that the at least one feature is indicative of whether content is malign; and
generating, based on the at least one feature, at least one feature vector representation of each training file of the plurality of training files indicating whether the malign at least one feature is present in each training file of the plurality of training files.
4. The method of claim 1 further comprising:
determining, via the second classifier and based on a file of the plurality of files being a truncated file, a third classification for the truncated file.
5. The method of claim 1 , further comprising:
determining, based on parsing a file header to determine an expected section length and determining that a total file buffer size is less than a sum of the section lengths, that a file of the plurality of files is a truncated file; and
determining, via the second classifier and based on the file being a truncated file, a third classification for the truncated file.
6. The method of claim 1 , wherein the first classification comprises at least one of malicious or benign, and wherein the second classification comprises at least one of malicious or benign.
7. The method of claim 1 , further comprising:
determining, based on execution of a first algorithm of a plurality of algorithms in a first stage, that a plurality of features from a plurality of training files are indicative of whether content is malign; and
determining, based on execution of a second algorithm of the plurality of algorithms in a second stage, a portion of the plurality of features that are associated with a highest success rate of indicating whether content is malign.
8. The method of claim 7 , wherein the plurality of algorithms comprises at least one of random selection, entropy-gain, minimum Redundancy Maximum Relevance (mRMR), or entropy-mask feature exclusion.
9. The method of claim 1 , determining, based on at least one feature from each training file of a plurality of training files, that the at least one feature is indicative of whether content is malign, wherein the determining that the at least one feature is indicative of whether content is malign comprises:
comparing entropy of the at least one feature to a threshold.
10. The method of claim 9 , wherein the threshold is configured based on information based on a user input.
11. The method of claim 1 , determining, based on at least one feature from each training file of a plurality of training files, that the at least one feature is indicative of whether content is malign, wherein the at least one feature comprises an n-gram comprising a sequence of n consecutive bytes.
12. A device comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the device to:
determine at least a first file-type and a second file-type associated with a plurality of files;
determine, via a first classifier of a plurality of classifiers that is trained to determine whether files of the first file-type are at least malign or benign, a first classification for a first file of the plurality of files, wherein the first file is of the first file-type; and
determine, via a second classifier of the plurality of classifiers that is trained to determine whether files of a plurality of file-types are at least malign or benign and based on a second file of the plurality of files being of the second file-type that is different than the first file-type, a second classification for the second file.
13. The device of claim 12 , wherein the determining at least the first file-type further comprises:
determining, based on at least one byte sequence at an offset in the first file, the first file-type; and
determining, based on the first file-type, whether an internal structure of the first file matches an expected internal structure.
14. The device of claim 12 , wherein training each classifier of the plurality of classifiers comprises:
determining, based on at least one feature from each training file of a plurality of training files, that the at least one feature is indicative of whether content is malign, wherein the at least one feature comprises an n-gram comprising a sequence of n consecutive bytes; and
generating, based on the at least one feature, at least one feature vector representation of each training file of the plurality of training files indicating whether the malign at least one feature is present in each training file of the plurality of training files.
15. The device of claim 14 , wherein training each classifier of the plurality of classifiers comprises:
determining, based on at least one feature from each training file of a plurality of training files, that the at least one feature is indicative of whether content is malign, wherein determining that the at least one feature is indicative of whether content is malign comprises:
comparing entropy of the at least one feature to a threshold, wherein the threshold is configured based on information based on a user input.
16. The device of claim 12 , wherein the instructions, when executed by the one or more processors, further cause the device to:
determine, via the second classifier and based on a file of the plurality of files being a truncated file, a third classification for the truncated file.
17. The device of claim 12 , wherein the instructions, when executed by the one or more processors, further cause the device to:
determine, based on parsing a file header to determine an expected section length and determining that a total file buffer size is less than a sum of the section lengths, that a file of the plurality of files is a truncated file; and
determine, via the second classifier and based on the file being a truncated file, a third classification for the truncated file.
18. The device of claim 12 , wherein the first classification comprises at least one of malicious or benign, and wherein the second classification comprises at least one of malicious or benign.
19. The device of claim 12 , wherein the instructions, when executed by the one or more processors, further cause the device to:
determine, based on execution of a first algorithm of a plurality of algorithms in a first stage, that a plurality of features from a plurality of training files are indicative of whether content is malign; and
determine, based on execution of a second algorithm of the plurality of algorithms in a second stage, a portion of the plurality of features that are associated with a highest success rate of indicating whether content is malign.
20. The device of claim 19 , wherein the plurality of algorithms comprises at least one of random selection, entropy-gain, minimum Redundancy Maximum Relevance (mRMR), or entropy-mask feature exclusion.
21. A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by a processor, cause:
determining at least a first file-type and a second file-type associated with a plurality of files;
determining, via a first classifier of a plurality of classifiers that is trained to determine whether files of the first file-type are at least malign or benign, a first classification for a first file of the plurality of files, wherein the first file is of the first file-type; and
determining, via a second classifier of the plurality of classifiers that is trained to determine whether files of a plurality of file-types are at least malign or benign and based on a second file of the plurality of files being of the second file-type that is different than the first file-type, a second classification for the second file.
22. The non-transitory computer-readable storage medium of claim 21 , wherein the determining at least the first file-type further comprises:
determining, based on at least one byte sequence at an offset in the first file, the first file-type; and
determining, based on the first file-type, whether an internal structure of the first file matches an expected internal structure.