Byte-distribution analysis of file security

1. A computer-based method for scanning files for security, comprising:

receiving, by a computer, an unfamiliar file for scanning;

generating, by the computer, a histogram of frequencies of occurrence of a plurality of byte values within a buffer of file data from the unfamiliar file, excluding frequencies for a designated set of byte values from the plurality of byte values; and

if the generated histogram of non-excluded frequencies deviates substantially from a reference frequency distribution, then signaling, by the computer, that the unfamiliar file is potentially malicious.

2. The method of claim 1 further comprising transforming, by the computer, the unfamiliar file prior to said generating a histogram.

3. The method of claim 2 wherein said transforming comprises decoding the unfamiliar file.

4. The method of claim 2 wherein said transforming comprises disassembling byte code in the unfamiliar file.

5. The method of claim 2 wherein said transforming comprises transforming the bytes of the unfamiliar file to transformed byte values.

6. The method of claim 1 further comprising determining, by the computer, if a mime type of the unfamiliar file is suitable for analysis.

7. The method of claim 1 wherein the reference frequency distribution is a uniform distribution.

8. The method of claim 7 further comprising calculating, by the computer, an average of the non-excluded frequencies within the buffer of file data, and wherein the histogram of non-excluded frequencies is deemed to deviate substantially from the reference uniform distribution if any of the non-excluded frequencies deviates substantially from the calculated average.

9. The method of claim 8 wherein a non-excluded frequency is deemed to deviate substantially from the calculated average if the frequency exceeds the product of a threshold multiplier and the average.

10. The method of claim 1 further comprising deriving, by the computer, the designated set of byte values whose frequencies are excluded from the histogram by processing a training set of non-malicious files.

11. A computer system for scanning files for security, comprising:

a computer processor; and

a memory unit storing:

instructions for a histogram generator, executed by said computer processor, for building a histogram of frequencies of occurrences of a plurality of byte values within a buffer of file data from an unfamiliar file, excluding frequencies of a designated set of byte values from the plurality of byte values; and

instructions for a threshold detector, executed by said computer processor, for detecting if the histogram of the non-excluded frequencies generated by said histogram generator deviates substantially from a reference frequency distribution.

12. The computer system of claim 11 wherein said memory unit further comprises instructions for a file pre-processor, executed by said computer processor, for transforming the unfamiliar file prior to said histogram generator building the histogram of frequencies.

13. The computer system of claim 12 wherein said file pre-processor decodes the unfamiliar file.

14. The computer system of claim 12 wherein said file pre-processor disassembles byte code in the unfamiliar file.

15. The computer system of claim 12 wherein said file pre-processor transforms the bytes of the unfamiliar file to transformed byte values.

16. The computer system of claim 11 wherein said memory unit further comprises instructions for a mime type filter, executed by said computer processor, for determining whether a mime type of the unknown file is of a type deemed suitable for byte-distribution analysis.

17. The computer system of claim 11 wherein the reference frequency distribution is a uniform distribution.

18. The computer system of claim 17 wherein said storage unit further comprises instructions for an average calculator, executed by said computer processor, for calculating an average of frequencies of occurrence of the plurality of bytes within the buffer of file data, excluding frequencies of the designated set of bytes, and wherein said threshold detector detects that the histogram of frequency values of the non-excluded bytes deviates substantially from the reference uniform distribution if any of the frequencies of the non-excluded bytes deviates substantially from the average calculated by said average calculator.

19. The computer system of claim 18 wherein said threshold detector detects that a frequency of a non-excluded byte deviates substantially from the calculated average if the frequency exceeds the product of a threshold multiplier and the average.

20. The computer system of claim 11 wherein said storage unit further comprises instructions for a training processor, executed by said computer processor, for deriving the designated set of byte values whose frequencies are excluded from the histogram by processing a training set of non-malicious files.

21. A non-transitory computer-readable storage medium storing program code for causing a computing device:

to receive an unfamiliar file for scanning;

to generate a histogram of frequencies of occurrence of a plurality of byte values within the buffer of file data, excluding frequencies for a designated set of byte values from the plurality of byte values; and

if the generated histogram of non-excluded frequencies deviates substantially from a reference frequency distribution, then to signal that the unfamiliar file is potentially malicious.