System and method for malware detection
Systems and methods for malware detection techniques, which detect malware by identifying the C&C communication between the malware and the remote host. In particular, the disclosed techniques distinguish between request-response transactions that carry C&C communication and request-response transactions of innocent traffic. Individual request-response transactions may be analyzed rather than entire flows, and fine-granularity features examined within the transactions. As such, these methods and systems are highly effective in distinguishing between malware C&C communication and innocent traffic, i.e., in detecting malware with high detection probability and few false alarms.
1. A method, comprising: monitoring, with a network probe, request-response transactions that are exchanged in a computer system without transmitting all of the request-response transactions of the computer system through the network probe;
discarding transactions from the monitored request-response transactions that access a predetermined number of most-frequently-accessed hosts;
extracting one or more subsets of the monitored request-response transactions, which are exchanged with one or more respective nodes in the computer system, the one or more subsets comprising request-response transactions that are exchanged between one or more clients and a given host and between one or more hosts and a given client;
evaluating a set of multiple different features over the request-response transactions in the subsets by estimating aggregated statistical properties of the set of multiple different features over the request-response transactions in the subsets, the set of multiple different features comprising a plurality of: repetitions of a Uniform Resource Identifier (URI) in given requests in which the URI is a random string, a given response not indicating a referrer, a content length in a given response being shorter than a certain threshold value, a user agent in a given request being shorter than a certain threshold value, a number of fields in a given request being smaller than a certain threshold value, or a returned content in a given response being an executable,
wherein the set of multiple different features includes at least one feature that comprises a characteristic of one or more underlying protocols used for transmitting the request-response transactions;
wherein a certain aggregate statistical property is evaluated over each of a plurality of different time periods; and
based on the evaluated features, identifying whether the request-response transactions in the subsets are exchanged with a malicious software in the nodes;
wherein identifying whether the request-response transactions in the subsets are exchanged with a malicious software comprises detecting that the malicious software runs in the given client.
2. The method according to claim 1 , wherein identifying whether the request-response transactions in the subsets are exchanged with a malicious software comprises detecting that the given host controls the malicious software.
3. The method according to claim 1 , wherein evaluating the set of multiple different features comprises determining the set of multiple different features over header fields of the request-response transactions.
4. The method according to claim 1 , wherein evaluating the set of multiple different features comprises determining the set of multiple different features over a predefined number of first content bytes at a beginning of the request-response transactions.
5. The method according to claim 1 , wherein identifying whether the request-response transactions are exchanged with the malicious software comprises checking whether the aggregated statistical property meets a malware detection criterion.
6. The method according to claim 5 , wherein the malware criterion distinguishes between a first statistical distribution in values of the set of multiple different features, which is indicative of the malicious software, and a second statistical distribution that is indicative of innocent traffic.
7. The method according to claim 5 , and comprising adaptively adjusting the malware detection criterion by providing feedback data into malware detection software in communication with the network probe.
8. The method according to claim 1 , wherein monitoring of the transactions and extraction of the subsets are performed by a first processor, and wherein evaluation of the set of multiple different features and identification of the malicious software are performed by a second processor separate from the first processor.
9. The method of claim 1 further comprising:
detecting distinct activity periods in the one or more subsets of the monitored request-response transactions between the given host and the one or more clients, wherein the distinct activity periods are separated by periods of inactivity, wherein the distinct activity periods set the plurality of different time periods, and wherein the certain aggregate statistical property is evaluated over each of the detected activity periods.
10. Apparatus, comprising: a network probe that monitors request-response transactions that are exchanged in network traffic on a computer system without having the network traffic pass directly through the network probe; and
discarding transactions from the monitored request-response transactions that access a predetermined number of most-frequently-accessed hosts;
a processor that executes instructions:
to extract one or more subsets of the monitored request-response transactions, which are exchanged with one or more respective nodes in the computer system, the one or more subsets comprise request-response transactions that are exchanged between one or more clients and a given host and between one or more hosts and a given client;
to evaluate a set of multiple different features over the request-response transactions in the subsets by estimating aggregated statistical properties of the set of multiple different features over the request-response transactions in the subsets, the set of multiple different features comprising a plurality of: repetitions of a Uniform Resource Identifier (URI) in given requests in which the URI is a random string, a given response not indicating a referrer, a content length in a given response being shorter than a certain threshold value; a user agent in a given request being shorter than a certain threshold value, a number of fields in a given request being smaller than a certain threshold value, or a returned content in a given response being an executable,
wherein the set of multiple different features includes at least one feature that comprises a characteristic of one or more underlying protocols used for transmitting the request-response transactions;
wherein a certain aggregate statistical property is evaluated over each of a plurality of different time periods; and
based on the evaluated set of multiple different features, to identify whether the request-response transactions in the one or more subsets are exchanged with a malicious software in the nodes;
wherein identifying whether the request-response transactions in the subsets are exchanged with a malicious software comprises detecting that the malicious software runs in the given client.
11. The apparatus according to claim 10 , wherein the processor executes instructions to detect that the given host controls the malicious software by identifying whether the request-response transactions in the subsets are exchanged with a malicious software.
12. The apparatus according to claim 10 , wherein the processor executes instructions to evaluate the set of multiple different features over header fields of the request-response transactions.
13. The apparatus according to claim 10 , wherein the processor executes instructions to evaluate the set of multiple different features over a predefined number of first content bytes at a beginning of the request-response transactions.
14. The apparatus according to claim 10 , wherein the processor executes instructions to identify whether the request-response transactions are exchanged with the malicious software by checking whether the aggregated statistical properties of the set of multiple different features meets a malware detection criterion.
15. The apparatus of claim 10 , wherein the processor executes instructions to detect distinct activity periods in the one or more subsets of the monitored request-response transactions between the given host and the one or more clients, wherein the distinct activity periods are separated by periods of inactivity, wherein the distinct activity periods set the plurality of different time periods, and wherein the certain aggregate statistical property is evaluated over each of the detected activity periods.