Methods and systems for encoding computer processes for malware detection
A method for encoding computer processes for malicious program detection. The method includes the steps of (a) randomly sampling a trace of system calls collected over a predetermined interval, each system call including context information and memory addresses for the function being monitored; (b) computing system address differences from the trace of system calls and retaining the computed values; (c) forming a group of n-grams (words) of retained differences of system addresses from the trace of system calls; (d) forming a series of process snippets, each process snippet including context information and the retained differences of system addresses; (e) transforming each process snippet to form a compact representation (process dot) comprising a pair of elements c, a , wherein c includes the context information and a is a sparse vector that encodes information derived from the group of n-grams; (f) forming clusters of compact representations; (g) obtaining clusters of compact representations from one or more malicious program-free computers; and (h) comparing the clusters formed in step (f) to those obtained in step (g) and determining the presence of malicious program from the comparison.
1. In a managed network of computers, a method for encoding computer processes for malicious program detection, comprising the steps of:
(a) randomly sampling a trace of system calls collected over an observation interval, each system call including context information and memory addresses for a function being monitored;
(b) computing system address differences from the trace of system calls and retaining computed values;
(c) forming a group of n-grams (words) of retained differences of system addresses from the trace of system calls;
(d) forming a series of process snippets, each process snippet including the context information and the retained differences of system addresses;
(e) transforming each process snippet to form a compact representation (process dot) comprising a pair of elements (c, a), wherein c includes the context information and a is a sparse vector that encodes information derived from the group of n-grams;
(f) forming clusters of compact representations;
(g) obtaining clusters of compact representations from one or more malicious program-free computers; and
(h) comparing the clusters formed in step (f) to the clusters obtained in step (g) and determining presence of a malicious program from the comparison.
2. The method of claim 1 , wherein the context information includes a machine identifier, a batch number for the sample, and a timestamp.
3. The method of claim 2 , further comprising the step of determining a frequency of occurrence for each n-gram (word) and creating a dictionary of zeta n-grams, D ζ , by using words that correlate with each other above a predetermined threshold.
4. The method of claim 3 , further comprising the steps of: selecting one or more of the zeta words to be anchor words; encoding in a vector a ζ , the probabilities of the anchor words, given that the process snippet contains one or more zeta words; creating a sparse vector a μ , that estimates the probabilities for the non-zeta words in the process snippet; and combining by direct weighted sum the vector a ζ and the parse vector a μ into the sparse vector a of the process dot:
a=b ζ a ζ ⊕b μ a μ with weights b ζ and b μ .
5. The method of claim 4 , further comprising the step of maintaining a white list of non-zeta words.
6. The method of claim 4 , further comprising the step of maintaining a white list of non-zeta words that are randomly selected among possible candidates.
7. The method of claim 4 , wherein each entry in the sparse vector a of the compact representation is proportional to a probability that a feature is observed given an execution environment.
8. The method of claim 1 , wherein a waiting period is chosen from a random exponential distribution before repeating the steps of claim 1 .
9. The method of claim 1 , wherein the observational interval is chosen from a random distribution where the mean of the distribution is optimized to increase the detection rate of a system while maintaining a CPU usage below an upper bound.
10. The method of claim 9 , wherein the observational interval is five seconds.
11. The method of claim 1 , wherein system-call information is sampled without altering any original component of any operating system.
12. The method of claim 1 , wherein the method consumes less than 5% of a CPU'S time.
13. A method for encoding computer processes for malicious program detection, comprising the steps of:
(a) randomly sampling a trace of system calls collected over a predetermined interval, each system call including context information and memory addresses for a function being monitored;
(b) computing system address differences from the trace of system calls and retaining computed values;
(c) forming a group of n-grams (words) of retained differences of system addresses from the trace of system calls;
(d) forming a series of process snippets, each process snippet including the context information and the retained differences of system addresses;
(e) transforming each process snippet to form a compact representation (process dot) comprising a pair of elements (c, a), wherein c includes the context information and a is a sparse vector that encodes information derived from the group of n-grams;
(f) forming clusters of compact representations; and
(g) comparing the clusters formed in step (f) to a library of malicious programs samples.
14. The method of claim 13 wherein a library of non-malicious program samples is obtained dynamically from other devices that are deemed free of malicious activity based on a majority rule.
15. The method of claim 13 , wherein the context information includes a machine identifier, a batch number for the sample, and a timestamp.
16. The method of claim 13 , wherein there is a waiting period chosen from a random exponential distribution before repeating the steps of claim 13 .
17. The method of claim 16 , further comprising the step of determining a frequency of occurrence for each n-gram (word) and creating a dictionary of zeta n-grams, D ζ , by using words that correlate with each other above a predetermined threshold.
18. The method of claim 17 , further comprising the steps of: selecting one or more of the zeta words to be anchor words; encoding in a vector a ζ , the probabilities of the anchor words, given that the process snippet contains one or more zeta words; creating a sparse vector a μ , that estimates the probabilities for the non-zeta words in the process snippet; and combining by direct weighted sum the vector a ζ and the sparse vector a μ into the sparse vector a of the process dot:
a=b ζ a ζ ⊕b μ a μ with weights b ζ and b μ .
19. The method of claim 18 , further comprising the step of maintaining a white list of non-zeta words.
20. The method of claim 18 , further comprising the step of maintaining a white list of non-zeta words that are randomly selected among possible candidates.
21. The method of claim 18 , wherein each entry in the sparse vector a of the compact representation is proportional to a probability that a feature is observed given an execution environment.
22. The method of claim 13 , wherein the predetermined interval is chosen from a random distribution where the mean of the distribution is optimized to increase a detection rate of a system while maintaining a CPU usage below an upper bound.
23. The method of claim 22 , wherein the predetermined interval is five seconds.
24. The method of claim 13 , wherein system-call information is sampled without altering any original component of any operating system.
25. The method of claim 13 , wherein the method consumes less than 5% of a CPU'S time.
26. The method of claim 13 where the computer processes are executing in a managed network of computers.
27. A system for detecting a malicious program comprising: at least one computer; a sensor installed on the at least one computer, the sensor structured and arranged to collect information on resource utilization of the at least one computer; and a machine learning daemon structured and arranged to receive bundles of information from the sensor and determine a probability that the computer is infected with a malicious program, wherein the sensor and machine learning daemon are structured and arranged to:
randomly sample a trace of system calls collected over a predetermined interval, each system call including context information and memory addresses for a function being monitored;
compute system address differences from the trace of system calls and retaining computed values;
form a group of n-grams (words) of retained differences of system addresses from the trace of system calls;
form a series of process snippets, each process snippet including the context information and the retained differences of system addresses;
transform each process snippet to form a compact representation (process dot) comprising a pair of elements (c, a), wherein c includes the context information and a is a sparse vector that encodes information derived from the group of n-grams;
form clusters of compact representations; and
compare the clusters formed to a library of malicious program samples.
28. The system of claim 27 , wherein the context information includes a machine identifier, a batch number for the sample, and a timestamp.
29. The system of claim 27 , wherein there is a waiting period chosen from a random exponential distribution before repeating the steps of claim 27 .
30. The system of claim 29 , further comprising the step of determining a frequency of occurrence for each n-gram (word) and creating a dictionary of zeta n-grams, D ζ , by using words that correlate with each other above a predetermined threshold.
31. The system of claim 30 , further comprising the steps of: selecting one or more of the zeta words to be anchor words; encoding in a vector a ζ , the probabilities of the anchor words, given that the process snippet contains one or more zeta words; creating a sparse vector a μ that estimates the probabilities for the non-zeta words in the process snippet; and combining by direct weighted sum the vector a ζ and the sparse vector a μ into the sparse vector a of the process dot:
a=b ζ a ζ ⊕b μ a μ with weights b ζ and b μ .
32. The system of claim 31 , further comprising the step of maintaining a white list of non-zeta words.
33. The method of claim 31 , further comprising the step of maintaining a white list of non-zeta words that are randomly selected among possible candidates.
34. The system of claim 31 , wherein each entry in the sparse vector a of the compact representation is proportional to a probability that a feature is observed given an execution environment.
35. The system of claim 27 , wherein the predetermined interval is chosen from a random distribution where the mean of the distribution is optimized to increase a detection rate of the system while maintaining CPU usage below an upper bound.
36. The system of claim 35 , wherein the predetermined interval is five seconds.
37. The system of claim 27 , wherein system-call information is sampled without altering any original component of any operating system.
38. The system of claim 27 , wherein the system is installed on a managed network of computers having at least one server, each computer having a sensor installed therein.
39. The system of claim 27 , wherein the machine learning daemon is installed on the at least one server.
40. The system of claim 27 , wherein communication between the sensor and the machine learning daemon can be queued until later re-established.