Generation and use of trained file classifiers for malware detection
View Patent ↗A method includes accessing information identifying multiple files and identifying classification data for the multiple files, where the classification data indicates, for a particular file of the multiple files, whether the particular file includes malware. The method also includes generating a sequence of entropy indicators for each of the multiple files, each entropy indicator of the sequence of entropy indicators for the particular file corresponding to a chunk of the particular file. The method further includes generating n-gram vectors for the multiple files, where the n-gram vector for the particular file indicates occurrences of groups of entropy indicators in the sequence of entropy indicators for the particular file. The method also includes generating and storing a file classifier using the n-gram vectors and the classification data as supervised training data.
1. A computing device comprising:
a memory configured to store instructions to generate a file classifier; and
a processor configured to execute the instructions from the memory to perform operations comprising:
accessing information identifying multiple files and identifying classification data for the multiple files, wherein the classification data indicates, for a particular file of the multiple files, whether the particular file includes malware;
generating a sequence of entropy indicators for each of the multiple files, each entropy indicator of the sequence of entropy indicators for the particular file corresponding to a chunk of the particular file;
generating n-gram vectors for the multiple files, wherein an n-gram vector for the particular file indicates occurrences of groups of entropy indicators in the sequence of entropy indicators for the particular file; and
generating and storing a file classifier using the n-gram vectors and the classification data as supervised training data, wherein the supervised training data includes a plurality of n-gram vectors for each file, the plurality of n-gram vectors for at least one file including a zero-skip n-gram vector indicating occurrences of groups of adjacent entropy indicators, and including at least one skip n-gram vector indicating occurrences of groups of non-adjacent entropy indicators.
2. The computing device of claim 1 , wherein the n-gram vectors include a Boolean vector indicating the occurrences of the groups of entropy indicators.
3. The computing device of claim 1 , wherein the n-gram vectors include a count of the occurrences of the groups of entropy indicators.
4. The computing device of claim 1 , wherein each group of entropy indicators of the groups of entropy indicators includes two or more entropy indicators.
5. The computing device of claim 1 , wherein the file classifier corresponds to a decision tree, a neural network, or a support vector machine.
6. The computing device of claim 1 , wherein the operations further comprise generating printable characters corresponding to each of the multiple files, and wherein the file classifier is generated further based on the printable characters.
7. A method comprising:
accessing information identifying multiple files and identifying classification data for the multiple files, wherein the classification data indicates, for a particular file of the multiple files, whether the particular file includes malware;
generating a sequence of entropy indicators for each of the multiple files, each entropy indicator of the sequence of entropy indicators for the particular file corresponding to a chunk of the particular file;
generating n-gram vectors for the multiple files, where an n-gram vector for the particular file indicates occurrences of groups of entropy indicators in the sequence of entropy indicators for the particular file; and
generating and storing a file classifier using the n-gram vectors and the classification data as supervised training data, wherein the supervised training data includes a plurality of n-gram vectors for each file, the plurality of n-gram vectors for at least one file including a zero-skip n-gram vector indicating occurrences of groups of adjacent entropy indicators, and including at least one skip n-gram vector indicating occurrences of groups of non-adjacent entropy indicators.
8. The method of claim 7 , wherein the multiple files include files with malware, files without malware, and corresponding classification data.
9. The method of claim 7 , wherein the n-gram vectors include a Boolean vector indicating the occurrences of the groups of entropy indicators.
10. The method of claim 7 , wherein the n-gram vector includes a count of the occurrences of the groups of entropy indicators.
11. The method of claim 7 , wherein a group of entropy indicators of the groups of entropy indicators corresponds to two adjacent entropy indicators.
12. The method of claim 7 , further comprising generating printable characters corresponding to each of the multiple files, and wherein the file classifier is generated further based on the printable characters.
13. The method of claim 7 , wherein generating the n-gram vectors includes generating, for each file, a plurality of n-gram vectors, the plurality of n-gram vectors for each file including a zero-skip n-gram vector indicating the occurrences of the groups of adjacent entropy indicators and including multiple n-gram vectors indicating occurrences of pairs of printable characters.
14. A computer-readable storage device storing instructions that, when executed, cause a computer to perform operations comprising:
accessing information identifying multiple files and identifying classification data for the multiple files, wherein the classification data indicates, for a particular file of the multiple files, whether the particular file includes malware;
generating a sequence of entropy indicators for each of the multiple files, each entropy indicator of the sequence of entropy indicators for the particular file corresponding to a chunk of the particular file;
generating n-gram vectors for the multiple files, wherein an n-gram vector for the particular file indicates occurrences of groups of entropy indicators in the sequence of entropy indicators for the particular file; and
generating and storing a file classifier using the n-gram vectors and the classification data as supervised training data, wherein the supervised training data includes a plurality of n-gram vectors for each file, the plurality of n-gram vectors for at least one file including a zero-skip n-gram vector indicating occurrences of groups of adjacent entropy indicators, and including at least one skip n-gram vector indicating occurrences of groups of non-adjacent entropy indicators.
15. The computer-readable storage device of claim 14 , wherein the multiple files include files with malware, files without malware, and corresponding classification data.
16. The computer-readable storage device of claim 14 , wherein the operations further comprise generating a plurality of printable characters representing the file, and wherein generating the n-gram vectors includes generating, for at least one file, an n-gram vector indicating occurrences of pairs of printable characters.
17. The computer-readable storage device of claim 16 , wherein generating the plurality of printable characters representing the file includes converting at least a portion of the file to American Standard Code for Information Interchange (ASCII) characters.
18. The computer-readable storage device of claim 14 , wherein the supervised training data includes the plurality of n-gram vectors for each file.
19. The computer-readable storage device of claim 18 , further including multiple n-gram vectors indicating occurrences of pairs of printable characters.