Machine learning-based malicious attachment detector
A method includes receiving, from a pre-processor, an output file, the output file having been created by the pre-processor in response to input of an electronic file to the pre-processor, the electronic file being an attachment to an electronic mail message that is in-transit to a recipient computer on a network, the electronic file being a spreadsheet file, the output file containing features that are created by the pre-processor; receiving, from a machine learning-based classifier, malware classification data, the malware classification data being output by the machine learning-based classifier in response to the machine learning-based classifier determining whether the features are indicators of obfuscation, the data used to create the machine learning-based classifier including output files previously created by the pre-processor; in response to the malware classification data matching a criterion, causing the network to modify, delay, or block transmission of the electronic file to the recipient computer.
1. A computer-implemented method comprising:
receiving, from a pre-processor, an output file, the output file having been created by the pre-processor in response to input of an electronic file to the pre-processor, the electronic file being an attachment to an electronic mail message that is in-transit to a recipient computer on a network, the electronic file being a spreadsheet file, the output file containing features that are created by the pre-processor analyzing one or more sub-features of the electronic file, including one or more of: a sheet count of the spreadsheet file, a word count from cells of the spreadsheet file, a character count from words in the cells of the spreadsheet file, and text entropy from text in the cells of the spreadsheet file;
receiving, from a machine learning-based classifier, malware classification data that indicates whether the electronic file does or does not contain malware, the malware classification data being output by the machine learning-based classifier in response to the machine learning-based classifier determining that the features are or are not indicators of obfuscation, the data used to create the machine learning-based classifier including output files previously created by the pre-processor; and
in response to the malware classification data matching a criterion, causing the network to modify, delay, or block transmission of the electronic file to the recipient computer.
2. The method of claim 1 , wherein the pre-processor performs any one or more of the following to create the output file: parsing the electronic file, pre-execution static analysis of the electronic file, and extracting script components from the electronic file.
3. The method of claim 1 , wherein the features comprises any one or more of the following: an occurrence count, an occurrence frequency, metadata, a string of interest, a warning, a classification, and a rule name.
4. The method of claim 1 , further comprising:
performing one or more of the following: reducing the features using mutual information criteria and transforming the features to a floating point vector.
5. The method of claim 1 , wherein a model used by the machine learning-based classifier is created using any one or more of the following: a tree-based algorithm, a random forest algorithm, a deep learning algorithm, a neural network, and a deep convolutional neural network.
6. The method of claim 1 , the features comprising any one or more of: sheet count, word count from all cells, character count from all words in all cells, editing time, text entropy from all text in all cells, overall entropy from all text in all cells and all field types, occurrence count for strings of interest, and occurrence count for field types of interest.
7. The method of claim 1 , comprising receiving the electronic file from a message transfer application on the network.
8. A computer program product comprising:
one or more non-transitory computer-readable storage media coupled to one or more processors, the one or more non-transitory computer-readable storage media storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving, from a pre-processor, an output file, the output file having been created by the pre-processor in response to input of an electronic file to the pre-processor, the electronic file being an attachment to an electronic mail message that is in-transit to a recipient computer on a network, the electronic file being a spreadsheet file, the output file containing features that are created by the pre-processor analyzing one or more sub-features of the electronic file, including one or more of: a sheet count of the spreadsheet file, a word count from cells of the spreadsheet file, a character count from words in the cells of the spreadsheet file, and text entropy from text in the cells of the spreadsheet file;
receiving, from a machine learning-based classifier, malware classification data that indicates whether the electronic file does or does not contain malware, the malware classification data being output by the machine learning-based classifier in response to the machine learning-based classifier determining that the features are or are not indicators of obfuscation, the data used to create the machine learning-based classifier including output files previously created by the pre-processor; and
in response to the malware classification data matching a criterion, causing the network to modify, delay, or block transmission of the electronic file to the recipient computer.
9. The computer program product of claim 8 , wherein the pre-processor performs any one or more of the following to create the output file: parsing the electronic file, pre-execution static analysis of the electronic file, and extracting script components from the electronic file.
10. The computer program product of claim 8 , wherein the instructions, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
performing one or more of the following: reducing the features using mutual information criteria and transforming the features to a floating point vector.
11. The computer program product of claim 8 , wherein a model used by the machine learning-based classifier is created using any one or more of the following: a tree-based algorithm, a random forest algorithm, a deep learning algorithm, a neural network, and a deep convolutional neural network.
12. The computer program product of claim 8 , the features comprising any one or more of: sheet count, word count from all cells, character count from all words in all cells, editing time, text entropy from all text in all cells, overall entropy from all text in all cells and all field types, occurrence count for strings of interest, and occurrence count for field types of interest.
13. The computer program product of claim 8 , the instructions, when executed by one or more processors, cause the one or more processors to perform operations comprising receiving the electronic file from a message transfer application on the network.
14. A computer-implemented method comprising:
receiving, from a pre-processor, a first output file, the first output file having been created by the pre-processor in response to input of a first electronic file to the pre-processor, the pre-processor creating the first output file by parsing and analyzing the first electronic file and extracting script components from the first electronic file, the first electronic file being a first attachment to a first electronic mail message that is in-transit to a first recipient computer on a network, the first output file containing features that are created by the pre-processor analyzing one or more sub-features of the first electronic file, the features comprising one or more of: an occurrence count, an occurrence frequency, metadata, a string of interest, a warning, a classification, and a rule name, and performing a feature reduction of a number of the features in the first output file by using mutual information criteria to merge a subset of the features that contain similar information;
receiving, from a machine learning-based classifier, malware classification data that indicates whether the first electronic file does or does not contain malware, the malware classification data being output by the machine learning-based classifier in response to the machine learning-based classifier determining that the features are or are not indicators of obfuscation, the data used to create the machine learning-based classifier including output files previously created by the pre-processor; and
in response to the malware classification data matching a criterion, causing the network to modify, delay, or block transmission of the first electronic file to the recipient computer.
15. The method of claim 14 , further comprising:
creating, by the pre-processor, a second output file based on input of a second electronic file, wherein the second output file indicates that the second electronic file contains Visual Basic for Application (VBA) code or programming code, wherein the second electronic file is a second attachment to a second message that is in-transit to a second recipient computer on the network, and wherein the second output file includes features that are created by the pre-processor analyzing one or more sub-features of the second electronic file, wherein the features comprises any one or more of the following: a word count, a 128 byte frequency, a 128 byte raw-count, a count of comment lines, a ratio of number of comment lines to code lines, a number of code lines, a length of document in characters, a count of VBA scripts, an average number of lines per VBA script, and a largest number of lines per VBA script;
receiving, from the machine learning-based classifier, malware classification data that indicates that the second electronic file includes malware; and
causing the network to modify, delay, or block transmission of the second electronic file to the second recipient computer.
16. The method of claim 14 , further comprising:
creating, by the pre-processor, a second output file based on input of a second electronic file, wherein the second output file indicates that the second electronic file is a Rich Text Format (RTF) file or a word processor-created document file that does not contain programming code, wherein the second electronic file is a second attachment to a second message that is in-transit to a second recipient computer on the network, and wherein the second output file includes features that are created by the pre-processor analyzing one or more sub-features of the second electronic file, wherein the features comprises any one or more of the following: a page count, a word count, a character count, an editing time, a classification, a warning, a text entropy, an overall entropy, a document size, parse tree data, a rule name, an indicator priority, and an indicator category;
receiving, from the machine learning-based classifier, malware classification data that indicates that the second electronic file includes malware; and
causing the network to modify, delay, or block transmission of the second electronic file to the second recipient computer.
17. The method of claim 14 , further comprising:
creating, by the pre-processor, a second output file based on input of a second electronic file, wherein the second output file indicates that the second electronic file is a Portable Document Format (PDF) file or an image file, wherein the second electronic file is a second attachment to a second message that is in-transit to a second recipient computer on the network, and wherein the second output file includes features that are created by the pre-processor analyzing one or more sub-features of the second electronic file, wherein the features comprises any one or more of the following: a version number, a number of colors, an occurrence count for a string of interest, a count of embedded objects, a minimum object identifier, a maximum object identifier, a trailer minimum size, a trailer maximum size, a count of trailer objects, a count of streams, a stream minimum size, a stream maximum size, a fraction of a file that is part of a stream, a total entropy, a non-stream entropy, a file size in bytes, a file size in bytes for non-stream parts, an end of file (EOF) count, a count for a top occurring variable, a count of xref tags, a count of startxref tags, a count of embedded Uniform Resource Locators (URLs), and a top occurring metadata item;
receiving, from the machine learning-based classifier, malware classification data that indicates that the second electronic file includes malware; and
causing the network to modify, delay, or block transmission of the second electronic file to the second recipient computer.
18. The method of claim 14 , further comprising:
creating, by the pre-processor, a second output file based on input of a second electronic file, wherein the second output file indicates that the second electronic file is a Portable Document Format (PDF) file, wherein the second electronic file is a second attachment to a second message that is in-transit to a second recipient computer on the network, and wherein the second output file includes features that are created by the pre-processor analyzing one or more sub-features of the second electronic file, wherein the machine learning-based classifier is created using a neural network and the malware classification data is produced by inputting a single image of the electronic file to the machine learning-based classifier;
receiving, from the machine learning-based classifier, malware classification data that indicates that the second electronic file includes malware; and
causing the network to modify, delay, or block transmission of the second electronic file to the second recipient computer.
19. The method of claim 14 , further comprising:
creating, by the pre-processor, a second output file based on input of a second electronic file, wherein the second output file indicates that the second electronic file is a Flash file or a Small Web Format (SWF) file or a multimedia file, wherein the second electronic file is a second attachment to a second message that is in-transit to a second recipient computer on the network, and wherein the second output file includes features that are created by the pre-processor analyzing one or more sub-features of the second electronic file, wherein the features comprises any one or more of the following: a version number, a file size, a count of scripts, a count for a word of interest, a decompiler message, a line count, a character count, a 128 byte frequency, and a 128 byte raw-count;
receiving, from the machine learning-based classifier, malware classification data that indicates that the second electronic file includes malware; and
causing the network to modify, delay, or block transmission of the second electronic file to the second recipient computer.