System and method for statistical analysis of comparative entropy
In accordance with one embodiment of the present disclosure, a method for determining the similarity between a first data set and a second data set is provided. The method includes performing an entropy analysis on the first and second data sets to produce a first entropy result, wherein the first data set comprises data representative of a first one or more computer files of known content and the second data set comprises data representative of a one or more computer files of unknown content; analyzing the first entropy result; and if the first entropy result is within a predetermined threshold, identifying the second data set as substantially related to the first data set.
1. At least one non-transitory machine readable storage medium, comprising computer-executable instructions carried on the computer readable storage medium, the instructions, when read and executed by a processor, cause the processor to:
electronically receive one or more files of an unknown content;
determine that the one or more files of an unknown content contain malware by:
comparing token values between a first probability distribution function associated with a first data set comprising one of more computer files of a first known content and a second probability distribution function associated with a second data set comprising the one or more computer files of an unknown content, wherein the first known content is known to be malware;
generating a first entropy result based on a difference between an expected number of occurrences of the token values in the first probability distribution function and an actual number of occurrences of the token values in the second probability distribution function; and
determining that the second data set is substantially related to the first data set based on a determination that the first entropy result is within a threshold; and
based on the determination that the one or more files of an unknown content contain malware, notify a user that the unknown content is malware.
2. The at least one non-transitory machine readable storage medium of claim 1 , further comprising instructions to cause the processor to:
compare the token values between a third probability distribution function of a third data set comprising a second known content and the second probability distribution function;
generate a second entropy result from the third data set and the second data set; and
determine whether the second data set is substantially related to the third data set based on whether the second entropy result is within the threshold.
3. The at least one non-transitory machine readable storage medium of claim 1 , wherein the instructions that cause the processor to generate the first entropy result further cause the processor to determine whether a first entropy value associated with the first data set is mathematically similar to a second entropy value associated with the second data set.
4. The at least one non-transitory machine readable storage medium of claim 1 , wherein the determination whether the second data is substantially related to the first data set is further based on whether the second data set is likely derived from the first data set.
5. The at least one non-transitory machine readable storage medium of claim 1 , wherein the first data set comprises data representative of a probability distribution function associated with one or more resources, the one or more resources constituting a portion of the first known content.
6. The at least one non-transitory machine readable storage medium of claim 1 , wherein the second data set comprises data representative of a probability distribution function associated with one or more resources, the one or more resources constituting a portion of the unknown content.
7. The at least one non-transitory machine readable storage medium of claim 1 , wherein the first known content and the unknown content are members of one or more of a plurality of categories of computer files.
8. The at least one non-transitory machine readable storage medium of claim 7 , wherein the instructions further cause the processor to categorize the unknown content into the one or more of the plurality of categories of computer files based substantially on the determination that the second data set is substantially related to the first data set.
9. The at least one non-transitory machine readable storage medium of claim 7 , wherein the plurality of categories of computer files include at least one of: malware, source code, image files, or object code.
10. The at least one non-transitory machine readable storage medium of claim 2 , wherein the instructions that cause the processor to generate the second entropy result further cause the processor to determine whether a third entropy value associated with the third data set is mathematically similar to a second entropy value associated with the second data set.
11. The at least one he non-transitory machine readable storage medium of claim 2 , wherein the determination whether the second data is substantially related to the third data set is further based on whether the second data set is likely derived from the third data set.
12. The at least one non-transitory machine readable storage medium of claim 2 , wherein the third data set comprises data representative of a probability distribution function associated with one or more resources, the one or more resources constituting a portion of the second known content.
13. The at least one non-transitory machine readable storage medium of claim 2 , wherein the second known content is a member of one or more of a plurality of categories of computer files.
14. The at least one non-transitory machine readable storage medium of claim 13 , wherein the instructions further cause the processor to categorize the second known content into the one or more of the plurality of categories of computer files based substantially on the determination that the second data set is substantially related to the third data set.
15. The at least one non-transitory machine readable storage medium of claim 13 , wherein the plurality of categories of computer files include at least one of: malware, source code, image files, or object code.
16. An electronic system for determining the similarity between a first data set and a second data set, the system comprising:
a computing device configured to:
electronically receive one or more files of an unknown content;
determine that the one or more files of an unknown content contain malware by:
comparing token values between a first probability distribution function associated with the first data set comprising one or more computer files of a first known content and a second probability distribution function associated with the second data set comprising one or more computer files of an unknown content, wherein the first known content is known to be malware;
generating a first entropy result based on a difference between an expected number of occurrences of the token values in the first probability distribution function and an actual number of occurrences of the token values in the second probability distribution function; and
determining that the second data set is substantially related to the first data set based on a determination that the first entropy result is within a threshold; and
based on the determination that the one or more files of an unknown content contain malware, notify a user that the unknown content is malware.
17. The electronic system of claim 16 , wherein the computing device is also configured to:
compare the token values between a third probability distribution function of a third data set comprising a second known content and the second probability distribution function;
generate a second entropy result from the third data set and the second data set; and
determine whether the second data set is substantially related to the third data set based on whether the second entropy result is within the threshold.
18. The electronic system of claim 16 , wherein the computing device is also configured to generate the first entropy result further based on a determination whether a first entropy value associated with the first data set is mathematically similar to a second entropy value associated with the second data set.
19. The electronic system of claim 16 , wherein the computing device is also configured to determine whether the second data is substantially related to the first data set further based on whether the second data set is likely derived from the first data set.
20. The electronic system of claim 16 , wherein the first data set comprises data representative of a probability distribution function associated with one or more resources, the one or more resources constituting a portion of the first known content.
21. The electronic system of claim 16 , wherein the second data set comprises data representative of a probability distribution function associated with one or more resources, the one or more resources constituting a portion of the unknown content.
22. The electronic system of claim 16 , wherein the first known content and the unknown content are members of one or more of a plurality of categories of computer files.
23. The electronic system of claim 22 , wherein the computing device is further configured to categorize the unknown content into the one or more of the plurality of categories of computer files based substantially on the determination that the second data set is substantially related to the first data set.
24. The electronic system of claim 22 , wherein the plurality of categories of computer files include at least one of: malware, source code, image files, or object code.
25. The electronic system of claim 17 , wherein the computing device is also configured to generate the second entropy result further based on a determination whether a third entropy value associated with the third data set is mathematically similar to a second entropy value associated with the second data set.
26. The electronic system of claim 17 , wherein the computing device is also configured to determine whether the second data is substantially related to the third data set further based on whether the second data set is likely derived from the third data set.
27. The electronic system of claim 17 , wherein the third data set comprises data representative of a probability distribution function associated with one or more resources, the one or more resources constituting a portion of the second known content.
28. The electronic system of claim 17 , wherein the second known content is a member of one or more of a plurality of categories of computer files.
29. The electronic system of claim 28 , wherein the computing device is further configured to categorize the second known content into the one or more of the plurality of categories of computer files based substantially on the determination that the second data set is substantially related to the third data set.
30. The electronic system of claim 28 , wherein the plurality of categories of computer files include at least one of: malware, source code, image files, or object code.