Systems and methods for executable code detection, automatic feature extraction and position independent code detection
Disclosed herein are systems and methods for enabling the automatic detection of executable code from a stream of bytes. In some embodiments, the stream of bytes can be sourced from the hidden areas of files that traditional malware detection solutions ignore. In some embodiments, a machine learning model is trained to detect whether a particular stream of bytes is executable code. Other embodiments described herein disclose systems and methods for automatic feature extraction using a neural network. Given a new file, the systems and methods may preprocess the code to be inputted into a trained neural network. The neural network may be used as a “feature generator” for a malware detection model. Other embodiments herein are directed to systems and methods for identifying, flagging, and/or detecting threat actors which attempt to obtain access to library functions independently.
1 . A computer-implemented method for programmatically identifying executable code within a file, the method comprising:
extracting, by a computer system, from a sequence of bytes of a file, a number of n-grams, wherein each n-gram comprises a series of bytes in the sequence of bytes;
generating, by the computer system, an array of counters, each counter of the array associated with one of the n-grams, wherein each counter comprises an integer value based on a frequency of occurrence of the associated n-gram within the sequence of bytes;
applying, by the computer system, a predictive model to the array of counters to determine a probability that the sequence of bytes comprises the executable code; and
flagging, by the computer system, the sequence of bytes of the file for further analysis by a malware detection system when the probability that the sequence of bytes comprises executable code is above a predetermined threshold,
wherein the computer system comprises a computer processor and an electronic storage medium.
2 . The computer-implemented method of claim 1 , further comprising accessing, by the computer system, the sequence of bytes from a portion of the file.
3 . The computer-implemented method of claim 2 , wherein the portion of the file comprises one or more of a resource, a string, a variable, an overlay, or a section, and wherein the portion of the file does not comprise executable permissions.
4 . The computer-implemented method of claim 1 , wherein the executable code is programmatically identified without executing the sequence of bytes on a computer system.
5 . The computer-implemented method of claim 1 , wherein the file comprises an executable file format and/or a portable executable (PE) file.
6 . The computer-implemented method of claim 1 , wherein the n-grams comprise a first set of n-grams with a first value of n and a second set of n-grams with a second value of n, wherein the predictive model uses the first set of n-grams and the second set of n-grams as input features to determine the probability of the sequence of bytes comprises executable code.
7 . The computer-implemented method of claim 1 , wherein n-grams comprise bi-grams, and wherein the bi-grams represent sequences of the two contiguous bytes within the sequence of bytes.
8 . The computer-implemented method of claim 1 , wherein the series of bytes of each respective n-gram comprises n number of bytes, wherein n is between 2 and 500.
9 . The computer-implemented method of claim 1 , wherein the number of n-grams corresponds to every n-gram present in the sequence of bytes.
10 . The computer-implemented method of claim 1 , wherein the series of bytes of each respective n-gram comprises n number of byte, wherein the n number of bytes in the series of bytes of each respective n-gram is selected based on the number of n-grams.
11 . The computer-implemented method of claim 1 , wherein the number of n-grams is a predetermined number between 50 and 10,000.
12 . The computer-implemented method of claim 1 , further comprising normalizing, by the computer system, each counter by a data length of the sequence of bytes.
13 . The computer-implemented method of claim 1 , wherein the predictive model comprises a plurality of models, each model of the plurality of models corresponding to a different machine architecture code.
14 . The computer-implemented method of claim 13 , wherein a machine architecture code comprises.NET, x86, and/or x64.
15 . The computer-implemented method of claim 1 , wherein the predictive model comprises at least one learning algorithm selected from the group of: support vector machines (SVM), linear regression, K-nearest neighbor (KNN) algorithm, logistic regression, naïve Bayes, linear discriminant analysis, decision trees, neural networks, or similarity learning.
16 . The computer-implemented method of claim 1 , wherein the predictive model comprises a random forest.
17 . The computer implemented method of claim 1 , wherein the random forest comprises a plurality of decision trees, each decision tree trained independently on a training set of bytes.
18 . A non-transitory computer readable medium containing program instructions for causing a computer to perform a method of:
extracting, from a sequence of bytes of a file, a number of n-grams, wherein each n-gram comprises a series of bytes in the sequence of bytes;
generating an array of counters, each counter of the array associated with one of the n-grams, wherein each counter comprises an integer value based on a frequency of occurrence of the associated n-gram within the sequence of bytes;
applying a predictive model to the array of counters to determine a probability that the sequence of bytes comprises the executable code; and
flagging the sequence of bytes of the file for further analysis by a malware detection system when the probability that the sequence of bytes comprises executable code is above a predetermined threshold.
19 . A computer system for programmatically identifying executable code within a file, the computer system comprising:
one or more computer readable storage devices configured to store a plurality of computer executable instructions; and
one or more hardware computer processors in communication with the one or more computer readable storage devices and configured to execute the plurality of computer executable instructions in order to cause the computer system to:
extract, from a sequence of bytes of a file, a number of n-grams, wherein each n-gram comprises a series of bytes in the sequence of bytes;
generate, an array of counters, each counter of the array associated with one of the n-grams, wherein each counter comprises an integer value based on a frequency of occurrence of the associated n-gram within the sequence of bytes;
apply, a predictive model to the array of counters to determine a probability that the sequence of bytes comprises the executable code; and
flag the sequence of bytes of the file for further analysis by a malware detection system when the probability that the sequence of bytes comprises executable code is above a predetermined threshold.