System and method for statistical analysis of comparative entropy
View Patent ↗In accordance with one embodiment of the present disclosure, a method for determining the similarity between a first data set and a second data set is provided. The method includes performing an entropy analysis on the first and second data sets to produce a first entropy result, wherein the first data set comprises data representative of a first one or more computer files of known content and the second data set comprises data representative of a one or more computer files of unknown content; analyzing the first entropy result; and if the first entropy result is within a predetermined threshold, identifying the second data set as substantially related to the first data set.
1. At least one non-transitory machine readable storage medium, comprising computer-executable instructions carried on the computer readable medium, the instructions readable by a processor, the instructions, when read and executed, for causing the processor to:
perform an entropy analysis on a first data set and a second data set to produce a first entropy result, wherein:
the first data set comprises data representative of a probability distribution function of token values associated with a first one or more computer files of known content and the second data set comprises data representative of a probability distribution function of token values associated with one or more computer files of unknown content; and
the entropy analysis includes causing the processor to:
compare token values between the probability distribution function of the computer files of known content and the probability distribution function of the computer files of unknown content;
generate the first entropy result based at least upon a difference between an expected number of occurrences of the token values in the probability distribution function of the computer files of known content and an actual number of occurrences of the token values in the probability distribution function of the computer files of unknown content;
based on a determination that the first entropy result is within a predetermined threshold, identify the second data set as substantially related to the first data set;
based upon identification of the second data set as substantially related to the first data set, identify malware resident on an electronic device.
2. The at least one non-transitory machine readable storage medium of claim 1 , further comprising instructions to cause the processor to:
perform the entropy analysis on a third data set and the second data set to produce a second entropy result, wherein the third data set comprises data representative of a probability distribution function associated with a second one or more computer files of known content; and
based on a determination that the second entropy result is within a predetermined threshold, identify the second data set as substantially related to the third data set.
3. The at least one non-transitory machine readable storage medium of claim 1 , wherein the instructions cause the processor to perform the entropy analysis on the first and second data sets by determining whether a first entropy value associated with the first data set is mathematically similar to a second entropy value associated with the second data set.
4. The at least one non-transitory machine readable storage medium of claim 2 , wherein the instructions cause the processor to perform the entropy analysis on the second and third data sets by determining whether a third entropy value associated with the third data set is mathematically similar to the second entropy value associated with the second data set.
5. The at least one non-transitory machine readable storage medium of claim 1 , wherein the instructions cause the processor to identify the second data as substantially related to the first data set by identifying the second data set as likely derived from the first data set.
6. The at least one non-transitory machine readable storage medium of claim 2 , wherein the instructions cause the processor to identify the second data as substantially related to the third data set by-identifying the second data set as likely derived from the third data set.
7. The at least one non-transitory machine readable storage medium of claim 1 , wherein the first data set comprises data representative of a probability distribution function associated with one or more resources, the one or more resources constituting a portion of the first one or more computer files of known content.
8. The at least one non-transitory machine readable storage medium of claim 1 , wherein the second data set comprises data representative of a probability distribution function associated with one or more resources, the one or more resources constituting a portion of the one or more computer files of unknown content.
9. The at least one non-transitory machine readable storage medium of claim 2 , wherein the third data set comprises data representative of a probability distribution function associated with one or more resources, the one or more resources constituting a portion of the second one or more computer files of known content.
10. The at least one non-transitory machine readable storage medium of claim 1 , wherein the first one or more computer files of known content and the one or more computer files of unknown content are members of one or more of a plurality of categories of computer files.
11. The at least one non-transitory machine readable storage medium of claim 2 , wherein the second one or more computer files of known content are members of the one or more of the plurality of categories of computer files.
12. The at least one non-transitory machine readable storage medium of claim 10 , wherein the instructions further cause the processor to categorize the one or more computer files of unknown content into the one or more of the plurality of categories of computer files based substantially on the identification of the second data set as substantially related to the first data set.
13. The at least one non-transitory machine readable storage medium of claim 11 , wherein the instructions further cause the processor to categorize the one or more computer files of unknown content into the one or more of the plurality of categories of computer files based substantially on the identification of the second data set as substantially related to the third data set.
14. The at least one non-transitory machine readable storage medium of claim 12 , wherein the instructions further cause the processor to:
based on a determination that the first entropy result is not within the predetermined threshold, expand the first data set to include additional data representative of the probability distribution function associated with the first one or more computer files of known content to create an expanded first data set; and
perform the entropy analysis on the expanded first data set and the second data set to produce a first refined entropy results;
analyze the first refined entropy result; and
based on a determination that the first refined entropy result is within the predetermined threshold, identify the second data set as substantially related to the first data set.
15. The at least one non-transitory machine readable storage medium of claim 13 , wherein the instructions further cause the processor to:
based on a determination that the second entropy result is not within the predetermined threshold, expand the third data set to include additional data representative of the probability distribution function associated with the second one or more computer files of known content to create an expanded third data set; and
perform the entropy analysis on the expanded third data set and the second data set to produce a second refined entropy results;
analyze the second refined entropy result; and
based on a determination that the second refined entropy result is within the predetermined threshold, identify the second data set as substantially related to the third data set.
16. The at least one non-transitory machine readable storage medium of claim 12 , wherein the plurality of categories of computer files include malware.
17. The at least one non-transitory machine readable storage medium of claim 13 , wherein the plurality of categories of computer files include malware.
18. The at least one non-transitory machine readable storage medium of claim 12 , wherein the plurality of categories of computer files include source code.
19. The at least one non-transitory machine readable storage medium of claim 13 , wherein the plurality of categories of computer files include source code.
20. The at least one non-transitory machine readable storage medium of claim 12 , wherein the plurality of categories of computer files include image files.
21. The at least one non-transitory machine readable storage medium of claim 13 , wherein the plurality of categories of computer files include image files.
22. The at least one non-transitory machine readable storage of claim 12 , wherein the plurality of categories of computer files include object code.
23. The at least one non-transitory machine readable storage medium of claim 13 , wherein the plurality of categories of computer files include object code.
24. The medium of claim 1 , further comprising instructions to, based upon identification of the second data set as substantially related to the first data set, take corrective action for malware.
25. The medium of claim 1 , further comprising instructions to, based upon identification of the second data set as substantially related to the first data set, identify an image resident on a computer.
26. An electronic system for determining the similarity between a first data set and a second data set, the system comprising:
a processor;
an entropy analysis engine comprising instructions to be executed by the processor, the instructions, when executed by the processor, configure the processor to perform an entropy analysis on a first data set and a second data set to produce a first entropy result, wherein the first data set comprises data representative of a probability distribution function of token values associated with a first one or more computer files of known content and the second data set comprises data representative of a probability distribution function of token values associated with one or more computer files of unknown content, the entropy analysis engine configured to analyze the first entropy result; and
a classification engine comprising instructions to be executed by the processor, the instructions, when executed by the processor, configure the classification engine to, based on a determination that the first entropy result is within a predetermined threshold, identify the second data set as substantially related to the first data set;
wherein the entropy analysis engine is further configured to:
compare token values between the probability distribution function of the computer files of known content and the probability distribution function of the computer files of unknown content; and
generate the first entropy result based at least upon a difference between an expected number of occurrences of the token values in the probability distribution function of the computer files of known content and an actual number of occurrences of the token values in the probability distribution function of the computer files of unknown content;
wherein the entropy analysis engine is further configured to, based upon identification of the second data set as substantially related to the first data set, identify malware resident on an electronic device.
27. The electronic system of claim 26 , wherein:
the entropy analysis engine is further configured to perform the entropy analysis on a third data set and the second dataset to produce a second entropy result, wherein the third data set comprises data representative of a probability distribution function associated with a second one or more computer files of known content; and
the classification engine is further configured to, based on a determination that the second entropy result is within a predetermined threshold, identify the second data set as substantially related to the third data set.
28. The electronic system of claim 26 , wherein the entropy analysis engine is configured to perform the entropy analysis on the first and second data sets by determining whether a first entropy value associated with the first data set is mathematically similar to a second entropy value associated with the second data set.
29. The electronic system of claim 27 , wherein the entropy analysis engine is further configured to perform the entropy analysis on the second and third data sets by determining whether a third entropy value associated with the third data set is mathematically similar to the second entropy value associated with the second data set.
30. The electronic system of claim 26 , wherein the classification engine is further configured to identify the second data as substantially related to the first data set by identifying the second data set as likely derived from the first data set.
31. The electronic system of claim 27 , wherein the classification engine is further configured to identify the second data as substantially related to the third data set by identifying the second data set as likely derived from the third data set.
32. The electronic system of claim 26 , wherein the first data set comprises data representative of a probability distribution function associated with one or more resources, the one or more resources constituting a portion of the first one or more computer files of known content.
33. The electronic system of claim 26 , wherein the second data set comprises data representative of a probability distribution function associated with one or more resources, the one or more resources constituting a portion of the one or more computer files of unknown content.
34. The electronic system of claim 27 , wherein the third data set comprises data representative of a probability distribution function associated with one or more resources, the one or more resources constituting a portion of the second one or more computer files of known content.
35. The electronic system of claim 26 , wherein the first one or more computer files of known content and the one or more computer files of unknown content are members of one or more of a plurality of categories of computer files.
36. The electronic system of claim 27 , wherein the second one or more computer files of known content are members of one or more of a plurality of categories of computer files.
37. The electronic system of Claim 35 , wherein the classification engine is further configured to categorize the one or more computer files of unknown content into the one or more of the plurality of categories of computer files based substantially on the identification of the second data set as substantially related to the first data set.
38. The electronic system of claim 36 , wherein the classification engine is further configured to categorize the one or more computer files of unknown content into the one or more of the plurality of categories of computer files based substantially on the identification of the second data set as substantially related to the third data set.
39. The electronic system of claim 37 , wherein:
the entropy analysis engine is further configured to:
based on a determination that the first entropy result is not within the predetermined threshold, expand the first data set to include additional data representative of the probability distribution function associated with the first one or more computer files of known content to create an expanded first data set; and
perform the entropy analysis on the expanded first data set and the second data set to produce a first refined entropy results; and
the classification engine is further configured to:
analyze the first refined entropy result; and
based on a determination that the first refined entropy result is within the predetermined threshold, identify the second data set as substantially related to the first data set.
40. The electronic system of claim 38 , wherein:
the entropy analysis engine is further configured to:
based on a determination that the second entropy result is not within the predetermined threshold, expand the third data set to include additional data representative of the probability distribution function associated with the second one or more computer files of known content to create an expanded third data set; and
perform the entropy analysis on the expanded third data set and the second data set to produce a second refined entropy results; and
the classification engine is further configured to:
analyze the second refined entropy result; and
based on a determination that the second refined entropy result is within the predetermined threshold, identify the second data set as substantially related to the third data set.
41. The electronic system of claim 38 , wherein the plurality of categories of computer files include malware.
42. The electronic system of claim 37 , wherein the plurality of categories of computer files include malware.
43. The electronic system of claim 38 , wherein the plurality of categories of computer files include source code.
44. The electronic system of claim 37 , wherein the plurality of categories of computer files include source code.
45. The electronic system of claim 38 , wherein the plurality of categories of computer files include image files.
46. The electronic system of claim 37 , wherein the plurality of categories of computer files include image files.
47. The electronic system of claim 38 , wherein the plurality of categories of computer files include object code.
48. The electronic system of claim 37 , wherein the plurality of categories of computer files include object code.
49. The system of claim 26 , wherein the entropy analysis engine is further configured to, based upon identification of the second data set as substantially related to the first data set, take corrective action for malware.
50. The system of claim 26 , wherein the entropy analysis engine is further configured to, based upon identification of the second data set as substantially related to the first data set, identify an image resident on a computer.