Machine learning-based malicious attachment detector
In an embodiment, a computer-implemented method includes receiving, from a pre-processor, an output file; where the output file is created by the pre-processor in response to input of an electronic file to the pre-processor; where the electronic file is an attachment to a message that is in-transit to a recipient computer on a network; where the output file contains features that are created by the pre-processor analyzing one or more sub-features of the electronic file; receiving, from a machine learning-based classifier, malware classification data that indicates whether the electronic file does or does not contain malware; where the malware classification data is output by the machine learning-based classifier in response to the machine learning-based classifier determining that the features are or are not indicators of obfuscation; where data used to create the machine learning-based classifier includes output files previously created by the pre-processor; in response to the malware classification data matching a criterion, causing the network to modify, delay, or block transmission of the electronic file to the recipient computer.
1. A computer-implemented method comprising:
receiving, from a pre-processor, a first output file;
wherein the first output file is created by the pre-processor in response to input of a first electronic file to the pre-processor;
wherein the first electronic file is an attachment to a first message that is in-transit to a first recipient computer on a network;
wherein the first output file contains features that are created by the pre-processor analyzing one or more sub-features of the first electronic file;
receiving, from a machine learning-based classifier, malware classification data that indicates whether the first electronic file does or does not contain malware;
wherein the malware classification data is output by the machine learning-based classifier in response to the machine learning-based classifier determining that the features are or are not indicators of obfuscation, wherein the indicators of obfuscation include one or more of:
a count of a blurred text exceeding a first threshold,
a percentage of the blurred text exceeding a second threshold, and
a count of embedded URLs exceeding a third threshold;
wherein data used to create the machine learning-based classifier includes output files previously created by the pre-processor; and
in response to the malware classification data matching a criterion, causing the network to modify, delay, or block transmission of the first electronic file to the recipient first computer.
2. The method of claim 1 , wherein the pre-processor performs any one or more of the following to create the first output file: parsing the first electronic file, pre-execution static analysis of the first electronic file, extracting script components from the first electronic file.
3. The method of claim 1 , wherein the features comprises any one or more of the following: an occurrence count, an occurrence frequency, metadata, a string of interest, a warning, a classification, a rule name.
4. The method of claim 1 , wherein the one or more sub-features of the first electronic file comprises any one or more of the following: a word, 128-byte data, a comment line, a line of code, a script, a page, a character, an edit, a version number, an image, a video, a color, a string of interest, an embedded object, an object identifier, a trailer object, a stream, an end of file (EOF) indicator, a variable, an xref tag, a startxref tag, a Uniform Resource Locators (URL), a metadata item, a version number, a file size, a decompiler message, a line of text.
5. The method of claim 1 , comprising any one or more of the following: reducing the features using mutual information criteria, transforming the features to a floating point vector.
6. The method of claim 1 , wherein a model used by the machine learning-based classifier is created using any one or more of the following: a tree-based algorithm, a random forest algorithm, a deep learning algorithm, a neural network, a deep convolutional neural network.
7. The method of claim 1 , further comprising:
creating, by the pre-processor, a second output file based on input of a second electronic file, wherein the second output file indicates that the electronic file contains Visual Basic for Application (VBA) code or programming code, wherein the second electronic file is a second attachment to a second message that is in-transit to a second recipient computer on the network, and wherein the second output file includes features that are created by the pre-processor analyzing one or more sub-features of the second electronic file, wherein the features comprises any one or more of the following: a word count, a 128-byte frequency, a 128 byte raw-count, a count of comment lines, a ratio of number of comment lines to code lines, a number of code lines, a length of document in characters, a count of VBA scripts, an average number of lines per VBA script, a largest number of lines per VBA script;
receiving, from the machine learning-based classifier, malware classification data that indicates that the second electronic file includes malware; and
causing the network to modify, delay, or block transmission of the second electronic file to the second recipient computer.
8. The method of claim 1 , further comprising:
creating, by the pre-processor, a second output file based on input of a second electronic file, wherein the second output file indicates that the second electronic file is a Rich Text Format (RTF) file or a word processor-created document file that does not contain programming code, wherein the second electronic file is a second attachment to a second message that is in-transit to a second recipient computer on the network, and wherein the second output file includes features that are created by the pre-processor analyzing one or more sub-features of the second electronic file, wherein the features comprises any one or more of the following: a page count, a word count, a character count, an editing time, a classification, a warning, a text entropy, an overall entropy, a document size, parse tree data, a rule name, an indicator priority, an indicator category;
receiving, from the machine learning-based classifier, malware classification data that indicates that the second electronic file includes malware; and
causing the network to modify, delay, or block transmission of the second electronic file to the second recipient computer.
9. The method of claim 1 , further comprising:
creating, by the pre-processor, a second output file based on input of a second electronic file, wherein the second output file indicates that the second electronic file is a Portable Document Format (PDF) file or an image file, wherein the second electronic file is a second attachment to a second message that is in-transit to a second recipient computer on the network, and wherein the second output file includes features that are created by the pre-processor analyzing one or more sub-features of the second electronic file, wherein the features comprises any one or more of the following: a version number, a number of colors, an occurrence count for a string of interest, a count of embedded objects, a minimum object identifier, a maximum object identifier, a trailer minimum size, a trailer maximum size, a count of trailer objects, a count of streams, a stream minimum size, a stream maximum size, a fraction of a file that is part of a stream, a total entropy, a non-stream entropy, a file size in bytes, a file size in bytes for non-stream parts, an end of file (EOF) count, a count for a top occurring variable, a count of xref tags, a count of startxref tags, a count of embedded Uniform Resource Locators (URLs), a top occurring metadata item;
receiving, from the machine learning-based classifier, malware classification data that indicates that the second electronic file includes malware; and
causing the network to modify, delay, or block transmission of the second electronic file to the second recipient computer.
10. The method of claim 1 , wherein the first output file indicates that the first electronic file is a Portable Document Format (PDF) file and the machine learning-based classifier is created using a neural network and the malware classification data is produced by inputting a single image of the first electronic file to the machine learning-based classifier.
11. The method of claim 1 , further comprising:
creating, by the pre-processor, a second output file based on input of a second electronic file, wherein the second output file indicates that the second electronic file is a Flash file or a Small Web Format (SWF) file or a multimedia file, wherein the second electronic file is a second attachment to a second message that is in-transit to a second recipient computer on the network, and wherein the second output file includes features that are created by the pre-processor analyzing one or more sub-features of the second electronic file, wherein the features comprises any one or more of the following: a version number, a file size, a count of scripts, a count for a word of interest, a decompiler message, a line count, a character count, a 128 byte frequency, a 128 byte raw-count;
receiving, from the machine learning-based classifier, malware classification data that indicates that the second electronic file includes malware; and
causing the network to modify, delay, or block transmission of the second electronic file to the second recipient computer.
12. The method of claim 1 , further comprising receiving the first electronic file from a message transfer application on the network.
13. A computer program product comprising:
one or more non-transitory computer-readable storage media storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving, from a pre-processor, a first output file;
wherein the first output file is created by the pre-processor in response to input of a first electronic file to the pre-processor;
wherein the first electronic file is an attachment to a first message that is in-transit to a first recipient computer on a network;
wherein the first output file contains features that are created by the pre-processor analyzing one or more sub-features of the first electronic file;
receiving, from a machine learning-based classifier, malware classification data that indicates whether the first electronic file does or does not contain malware;
wherein the malware classification data is output by the machine learning-based classifier in response to the machine learning-based classifier determining that the features are or are not indicators of obfuscation, wherein the indicators of obfuscation include one or more of:
a count of a blurred text exceeding a first threshold,
a percentage of the blurred text exceeding a second threshold, and
a count of embedded URLs exceeding a third threshold;
wherein data used to create the machine learning-based classifier includes output files previously created by the pre-processor; and
in response to the malware classification data matching a criterion, causing the network to modify, delay, or block transmission of the first electronic file to the recipient first computer.
14. The computer program product of claim 13 , wherein the pre-processor performs any one or more of the following to create the first output file: parsing the first electronic file, pre-execution static analysis of the first electronic file, extracting script components from the first electronic file.
15. The computer program product of claim 13 , wherein the features comprises any one or more of the following: an occurrence count, an occurrence frequency, metadata, a string of interest, a warning, a classification, a rule name.
16. The computer program product of claim 13 , wherein the one or more sub-features of the first electronic file comprises any one or more of the following: a word, 128-byte data, a comment line, a line of code, a script, a page, a character, an edit, a version number, an image, a video, a color, a string of interest, an embedded object, an object identifier, a trailer object, a stream, an end of file (EOF) indicator, a variable, an xref tag, a startxref tag, a Uniform Resource Locators (URL), a metadata item, a version number, a file size, a decompiler message, a line of text.
17. The computer program product of claim 13 , wherein the instructions, when executed by one or more processors, cause the one or more processors to perform operations comprising any one or more of the following: reducing the features using mutual information criteria, transforming the features to a floating point vector.
18. The computer program product of claim 13 , wherein a model used by the machine learning-based classifier is created using any one or more of the following: a tree-based algorithm, a random forest algorithm, a deep learning algorithm, a neural network, a deep convolutional neural network.
19. The computer program product of claim 13 , wherein the operations further comprise:
creating, by the pre-processor, a second output file based on input of a second electronic file, wherein the second output file indicates that the electronic file contains Visual Basic for Application (VBA) code or programming code, wherein the second electronic file is a second attachment to a second message that is in-transit to a second recipient computer on the network, and wherein the second output file includes features that are created by the pre-processor analyzing one or more sub-features of the second electronic file, wherein the features comprises any one or more of the following: a word count, a 128-byte frequency, a 128 byte raw-count, a count of comment lines, a ratio of number of comment lines to code lines, a number of code lines, a length of document in characters, a count of VBA scripts, an average number of lines per VBA script, a largest number of lines per VBA script;
receiving, from the machine learning-based classifier, malware classification data that indicates that the second electronic file includes malware; and
causing the network to modify, delay, or block transmission of the second electronic file to the second recipient computer.
20. The computer program product of claim 13 , wherein the operations further comprise:
creating, by the pre-processor, a second output file based on input of a second electronic file, wherein the second output file indicates that the second electronic file is a Rich Text Format (RTF) file or a word processor-created document file that does not contain programming code, wherein the second electronic file is a second attachment to a second message that is in-transit to a second recipient computer on the network, and wherein the second output file includes features that are created by the pre-processor analyzing one or more sub-features of the second electronic file, wherein the features comprises any one or more of the following: a page count, a word count, a character count, an editing time, a classification, a warning, a text entropy, an overall entropy, a document size, parse tree data, a rule name, an indicator priority, an indicator category;
receiving, from the machine learning-based classifier, malware classification data that indicates that the second electronic file includes malware; and
causing the network to modify, delay, or block transmission of the second electronic file to the second recipient computer.
21. The computer program product of claim 13 , wherein the operations further comprise:
creating, by the pre-processor, a second output file based on input of a second electronic file, wherein the second output file indicates that the second electronic file is a Portable Document Format (PDF) file or an image file, wherein the second electronic file is a second attachment to a second message that is in-transit to a second recipient computer on the network, and wherein the second output file includes features that are created by the pre-processor analyzing one or more sub-features of the second electronic file, wherein the features comprises any one or more of the following: a version number, a number of colors, an occurrence count for a string of interest, a count of embedded objects, a minimum object identifier, a maximum object identifier, a trailer minimum size, a trailer maximum size, a count of trailer objects, a count of streams, a stream minimum size, a stream maximum size, a fraction of a file that is part of a stream, a total entropy, a non-stream entropy, a file size in bytes, a file size in bytes for non-stream parts, an end of file (EOF) count, a count for a top occurring variable, a count of xref tags, a count of startxref tags, a count of embedded Uniform Resource Locators (URLs), a top occurring metadata item;
receiving, from the machine learning-based classifier, malware classification data that indicates that the second electronic file includes malware; and
causing the network to modify, delay, or block transmission of the second electronic file to the second recipient computer.
22. The computer program product of claim 13 , wherein the first output file indicates that the first electronic file is a Portable Document Format (PDF) file and the machine learning-based classifier is created using a neural network and the malware classification data is produced by inputting a single image of the first electronic file to the machine learning-based classifier.
23. The computer program product of claim 13 , wherein the operations further comprise:
creating, by the pre-processor, a second output file based on input of a second electronic file, wherein the second output file indicates that the second electronic file is a Flash file or a Small Web Format (SWF) file or a multimedia file, wherein the second electronic file is a second attachment to a second message that is in-transit to a second recipient computer on the network, and wherein the second output file includes features that are created by the pre-processor analyzing one or more sub-features of the second electronic file, wherein the features comprises any one or more of the following: a version number, a file size, a count of scripts, a count for a word of interest, a decompiler message, a line count, a character count, a 128 byte frequency, a 128 byte raw-count;
receiving, from the machine learning-based classifier, malware classification data that indicates that the second electronic file includes malware; and
causing the network to modify, delay, or block transmission of the second electronic file to the second recipient computer.
24. The computer program product of claim 13 , further comprising receiving the first electronic file from a message transfer application on the network.