Detection of stealthy malware activities with traffic causality and scalable triggering relation discovery

1. A computer system for distinguishing user-initiated network traffic from malware-initiated network traffic, the system comprising:

at least one central processing unit (CPU);

memory communicatively coupled to the CPU, the memory comprising program code executable by said at least one CPU to perform the following steps:

monitoring a plurality of unknown individual network events and identifying a traffic dependency among them, said dependency used to detect anomalies by confirming the legitimacy of the mot-trigger events;

converting a plurality of individual network events and a plurality of root-trigger events into a plurality of event pairs with comparable pairwise attributes, said pairwise attributes comprising one of the following: similarities in event's time stamps, domain names, host names, IP addresses, process IDs (PID), HTTP request's referral fields, network requests types, network requests content, network requests flags, to determine if there is a parent-child relationship between said one network event and one of its subsequent events wherein said root-trigger events are said parent, said individual network events are said child and said parent and said child are not identical;

said parent-child relationships between said individual network events and said root-trigger events are unknown when said individual network events and said root-trigger events are converted into event pairs;

determining for an individual child network event whether said individual child event has a legitimate parent root-trigger event; and

identifying malware-initiated traffic as an individual network event that does not have a legitimate parent root-trigger event.

2. The system of claim 1 wherein said system uses a pairing algorithm to extract the pairwise features and conduct learning classifiers to identify the dependency.

3. The system of claim 2 wherein said the machine learning classifier is a binary classification method.

4. The system of claim 1 wherein said the pairing algorithm includes a dictionary comprised of key-value pairs used to store a current network event,

wherein said key of said dictionary is the domain attribute of an event and said value is a set of requests, whose domain attribute is the same as said key; and

an event with an unmatched key value is filtered out.

5. The system of claim 1 wherein a feature extraction method is used to transform two individual network events into comparable pairwise attributes which are processed by learning algorithms.

6. The system of claim 5 wherein a binary classification method is used to discover whether a triggering relation exists in an event pair or not.

7. The system of claim 1 wherein a triggering relation graph is used to identify events that do not have any valid root triggers events.

8. The system of claim 7 wherein said triggering relation graph is used to detect anomalous or malicious network activities.

9. The system of claim 1 wherein one or more cost matrices are defined to penalize missed parent-child relationships.

10. The system of claim 9 wherein said one or more cost matrices are defined to weigh false positives and false negatives differently.

11. The system of claim 10 wherein said one or more cost matrices are defined to penalize false negatives 10 times more than false positives.

12. The system of claim 1 wherein malware-initiated traffic is identified as an individual network event that does not have a legitimate parent root-trigger by iteratively pairing an individual child network event with a plurality of parent root-trigger events.

13. The system of claim 1 wherein said system uses a pairing algorithm to extract the pairwise features, then applies pre-defined rules to identify the parent-child relationships of new events based on their pairwise features.

14. A method for creating a model to detect malware comprising the steps of:

obtaining network data comprising individual network events and root-trigger events, wherein said root-trigger events are user input events that occur prior in time to said individual network events;

performing a pairing operation on two or more of said individual network events and at least one said root-trigger events to transform said individual network events and said root-trigger events into event pairings having comparable pairwise attributes, said pairwise attributes comprising a one-parent-multiple-children relationship between said root-trigger events and said two or more individual network events, wherein said root-trigger events are said parent, said individual network events are said children and said parent and said children are not identical;

said individual network events and said root-trigger events are unknown when said individual network events and said root-trigger events are converted into event pairings; and

said parent-child relationships or other lineage relationships between said individual network events and said root-trigger events are unknown when said individual network events and said root-trigger events are converted into event pairings;

building a classification model that is used to identify malware-initiated traffic as an individual network event that does not have a legitimate root-trigger event.

15. The method of claim 14 wherein one or more binary classification algorithms are used.

16. The method of claim 14 wherein a root-trigger event is paired with a first network event and said first network event is paired with a second network event.

17. The method of claim 1 wherein said parent and said children form a branch.

18. The method of claim 14 wherein said parent and children form a chain.

19. The method of claim 14 wherein said parent and children form a chain comprised of a grandparent-parent-child relationship.