Methods and apparatus for machine learning based malware detection
Apparatus and methods describe herein, for example, a process that can include receiving a potentially malicious file, and dividing the potentially malicious file into a set of byte windows. The process can include calculating at least one attribute associated with each byte window from the set of byte windows for the potentially malicious file. In such an instance, the at least one attribute is not dependent on an order of bytes in the potentially malicious file. The process can further include identifying a probability that the potentially malicious file is malicious, based at least in part on the at least one attribute and a trained threat model.
1. A method, comprising:
receiving a target file;
dividing the target file into a plurality of byte windows;
for each byte window from the plurality of byte windows, determining (1) whether an informational entropy for that byte window is within an informational entropy range and (2) whether a byte standard deviation value for that byte window is within a byte standard deviation range;
calculating a probability that the target file is malicious based on the determining;
communicating a threat score based on the probability, the threat score associated with at least one of a potential threat to a user, a potential threat to a device, or a potential threat to a network; and
performing a remedial action based on the threat score.
2. The method of claim 1 , wherein the calculating includes calculating the probability using a trained threat model including at least one of a random forest classifier or a deep neural network.
3. The method of claim 1 , wherein the remedial action includes at least one of quarantining the target file, deleting the target file, sending a notification regarding the target file, cleaning the target file, or allowing the target file to execute within a virtual container.
4. The method of claim 1 , wherein the determining includes determining how often a combination of a string length range and a string hash value range occurs in each byte window from the plurality of byte windows.
5. The method of claim 1 , wherein the determining includes, for each byte window from the plurality of byte windows, determining how often a byte value in that byte window is within a byte value range.
6. The method of claim 1 , wherein the calculating includes calculating the probability using at least one of a rate of false positives associated with a trained threat model or a rate of false negatives associated with the trained threat model.
7. An apparatus, comprising:
a memory; and
a hardware processor operatively coupled to the memory,
the hardware processor configured to receive a target file, the hardware processor configured to calculate an attribute associated with the target file, the attribute based on, for each byte window from a plurality of byte windows of the target file, whether an informational entropy associated with that byte window is within an informational entropy range,
the hardware processor further configured to calculate, based on the attribute, a probability that the target file is malicious, and output a threat score based on the probability.
8. The apparatus of claim 7 , wherein the attribute is further based on, for each byte window from the plurality of byte windows of the target file, whether a byte standard deviation value for that byte window is within a byte standard deviation range.
9. The apparatus of claim 7 , wherein the hardware processor is configured to calculate the probability using a trained threat model including at least one of a random forest classifier or a deep neural network.
10. The apparatus of claim 7 , wherein the hardware processor is configured to calculate the threat score based on a security indication associated with a network associated with the target file.
11. The apparatus of claim 7 , wherein the hardware processor is configured to calculate the threat score using metadata associated with the target file.
12. The apparatus of claim 7 , wherein the attribute is further based on an indication of how often a combination of a string length range and a string hash value range occurs in each byte window from the plurality of byte windows.
13. The apparatus of claim 7 , wherein the attribute is further based on, for each byte window from the plurality of byte windows of the target file, how often a byte value in that byte window is within a byte value range.
14. A non-transitory processor-readable medium storing code representing instructions to be executed by a processor, the instructions comprising code to cause the processor to:
receive a target file;
determine an attribute associated with the target file, the attribute based on, for each byte window from a plurality of byte windows of the target file, whether a byte standard deviation value for that byte window is within a byte standard deviation range;
determine a probability that the target file is malicious based on the attribute; and
output a threat score based on the probability.
15. The non-transitory processor-readable medium of claim 14 , wherein the attribute is based on, for each byte window from the plurality of byte windows of the target file, whether an informational entropy associated with that byte window is within an informational entropy range.
16. The non-transitory processor-readable medium of claim 14 , wherein the code to cause the processor to determine the probability includes code to cause the processor to determine the probability that the target file is malicious using a trained threat model including at least one of a random forest classifier or a deep neural network classifier.
17. The non-transitory processor-readable medium of claim 14 , wherein the code to cause the processor to determine the probability includes code to cause the processor to determine the probability using at least one of a rate of false positives associated with a trained threat model or a rate of false negatives associated with the trained threat model.
18. The non-transitory processor-readable medium of claim 14 , wherein the code to cause the processor to determine the probability includes code to cause the processor to determine the probability that the target file is malicious using metadata associated with the target file.
19. The non-transitory processor-readable medium of claim 14 , wherein the target file is a Portable Executable (PE) file including PE fields, the code to cause the processor to determine the probability includes code to cause the processor to determine the probability based on information identified from the PE fields of the PE file.
20. The non-transitory processor-readable medium of claim 14 , wherein the attribute is based on, for each byte window from the plurality of byte windows of the target file, determining (1) how often a byte value in that byte window is within a byte value range and (2) whether an informational entropy associated with that byte window is within an informational entropy range.