Inline malware detection
Detection of malicious files is disclosed. A set comprising a plurality of sample classification models is received and stored. A determination is made that n-gram analysis should be performed on a sequence of received packets associated with a received file. Performing the n-gram analysis includes using a determined filetype associated with the sequence of received packets to select at least one stored sample classification model included in the set for use in performing the n-gram analysis. A determination is made that the received file is malicious based at least in part on the n-gram analysis of the sequence of received packets. In response to determining that the file is malicious, propagation of the received file is prevented.
1. A system, comprising:
a processor configured to:
receive and store a set comprising a plurality of sample classification models;
receive an indication that a file, having an associated transmitting device, and intended for a transmittee, is being transmitted as part of a network session comprising a sequence of packets, including by using a protocol decoder to detect a start of the file in a first portion of the sequence of packets of the network session;
perform n-gram analysis on a second portion of the sequence of received packets of the network session, wherein performing the n-gram analysis includes using a determined filetype associated with the sequence of received packets to select at least one stored sample classification model included in the set for use in performing the n-gram analysis, and wherein the n-gram analysis continues until a predetermined end-of-file criteria is met, wherein the criteria comprises at least one of: (1) an end-of-file context previously marked by the protocol decoder or (2) a predetermined, purported end-of-file is reached; and
determine, during the performing of the n-gram analysis on the second portion of the sequence of received packets of the network session, that the file is malicious based at least in part on the n-gram analysis of the second portion of the sequence of received packets, and in response to determining that the file is malicious, take a remedial action, including against the associated transmitting device;
wherein the maliciousness determination is made, during active transmission of the file transmission and before the file transmission is complete, without requiring that the entire sequence of received packets be subjected to n-gram analysis; and
a memory coupled to the processor and configured to provide the processor with instructions.
2. The system of claim 1 , wherein the processor is configured to perform the n-gram analysis at least in part by comparing n-grams in the received packets against a predetermined list of n-grams.
3. The system of claim 2 , wherein the predetermined list of n-grams was generated using a plurality of previously collected malware samples.
4. The system of claim 1 , wherein the processor is configured to select, from the plurality of sample classification models, a linear classification model, based on the determined filetype associated with the sequence of received packets.
5. The system of claim 1 , wherein performing the n-gram analysis includes accumulating a set of weights corresponding to observed n-grams.
6. The system of claim 5 , wherein the set of weights are accumulated in a single float value.
7. The system of claim 1 , wherein the processor is configured to select, from the plurality of sample classification models, a non-linear classification model, based on the determined filetype associated with the sequence of received packets.
8. The system of claim 7 , wherein the non-linear classification model includes n-gram features and non n-gram features.
9. The system of claim 8 , wherein at least one non n-gram feature is associated with a file size.
10. The system of claim 8 , wherein at least one non n-gram feature is associated with a presence of an overlay.
11. The system of claim 1 , wherein performing the n-gram analysis includes updating a value for a feature in a feature vector whenever the feature is matched.
12. The system of claim 1 , wherein the end-of-file context is a purported file length that is not an actual file length and the determination is made prior to reaching an actual end of the file.
13. The system of claim 1 , wherein the processor is further configured to receive at least one updated classification model.
14. The system of claim 1 , wherein the n-gram analysis is performed inline with other types of packet analyses as a single pass analysis.
15. The system of claim 1 , wherein the processor is further configured to use a set of whitelisted n-grams when performing the n-gram analysis.
16. The system of claim 1 , wherein the processor is further configured to transmit a copy of the file to a security platform and perform the n-gram analysis while awaiting a verdict from the security platform.
17. The system of claim 1 , wherein the remedial action comprises terminating the session.
18. The system of claim 1 , wherein the remedial action comprises placing the file in a quarantine area.
19. A method for evaluating and selectively blocking network traffic, comprising:
receiving and storing a set comprising a plurality of sample classification models;
receiving an indication that a file, having an associated transmitting device, and intended for a transmittee, is being transmitted as part of a network session comprising a sequence of packets, including by using a protocol decoder to detect a start of the file in a first portion of the sequence of packets of the network session;
performing n-gram analysis on a second portion of the sequence of received packets of the network session, wherein performing the n-gram analysis includes using a determined filetype associated with the sequence of received packets to select at least one stored sample classification model included in the set for use in performing the n-gram analysis, and wherein the n-gram analysis continues until a predetermined end-of-file criteria is met, wherein the criteria comprises at least one of: (1) an end-of file context previously marked by the protocol decoder or (2) a predetermined, purported end-of-file is reached; and
determining, during the performing of the n-gram analysis on the second portion of the sequence of received packets of the network session, that the file is malicious based at least in part on the n-gram analysis of the second portion of the sequence of received packets, and in response to determining that the file is malicious, taking a remedial action, including against the associated transmitting device;
wherein the maliciousness determination is made, during active transmission of the file transmission and before the file transmission is complete, without requiring that the entire sequence of received packets be subjected to n-gram analysis.
20. The method of claim 19 , wherein performing the n-gram analysis includes comparing n-grams in the received packets against a predetermined list of n-grams.
21. The method of claim 20 , wherein the predetermined list of n-grams was generated using a plurality of previously collected malware samples.
22. The method of claim 19 , wherein selecting the at least one stored sample classification model includes selecting a linear classification model based on the determined filetype associated with the sequence of received packets.
23. The method of claim 19 , wherein performing the n-gram analysis includes accumulating a set of weights corresponding to observed n-grams.
24. The method of claim 23 , wherein the set of weights are accumulated in a single float value.
25. The method of claim 19 , wherein selecting the at least one stored sample classification model includes selecting a non-linear classification model based on the determined filetype associated with the sequence of received packets.
26. The method of claim 25 , wherein the non-linear classification model includes n-gram features and non n-gram features.
27. The method of claim 26 , wherein at least one non n-gram feature is associated with a file size.
28. The method of claim 26 , wherein at least one non n-gram feature is associated with a presence of an overlay.
29. The method of claim 19 , wherein performing the n-gram analysis includes updating a value for a feature in a feature vector whenever the feature is matched.
30. The method of claim 19 , wherein the end-of-file context is a purported file length that is not an actual file length and the determination is made prior to reaching an actual end of the file.
31. The method of claim 19 , further comprising receiving at least one updated classification model.
32. The method of claim 19 , wherein the n-gram analysis is performed inline with other types of packet analyses as a single pass analysis.
33. The method of claim 19 , further comprising using a set of whitelisted n-grams when performing the n-gram analysis.
34. The method of claim 19 , further comprising transmitting a copy of the received file to a security platform and performing the n-gram analysis while awaiting a verdict from the security platform.
35. The method of claim 19 , wherein the remedial action comprises terminating the session.
36. The method of claim 19 , wherein the remedial action comprises placing the file in a quarantine area.
37. A computer program product comprising non-transitory computer readable storage medium and comprising computer instructions for:
receiving and storing a set comprising a plurality of sample classification models;
receiving an indication that a file, having an associated transmitting device, and intended for a transmittee, is being transmitted as part of a network session comprising a sequence of packets, including by using a protocol decoder to detect a start of the file in a first portion of the sequence of packets of the network session;
performing n-gram analysis on a second portion of the sequence of received packets of the network session, wherein performing the n-gram analysis includes using a determined filetype associated with the sequence of received packets to select at least one stored sample classification model included in the set for use in performing the n-gram analysis, and wherein the n-gram analysis continues until a predetermined end-of-file criteria is met, wherein the criteria comprises at least one of: (1) an end-of-file context previously marked by the protocol decoder or (2) a predetermined, purported end-of-file is reached; and
determining, during the performing of the n-gram analysis on the second portion of the sequence of received packets of the network session, that the file is malicious based at least in part on the n-gram analysis of the second portion of the sequence of received packets, and in response to determining that the file is malicious, taking a remedial action, including against the associated transmitting device;
wherein the maliciousness determination is made, during active transmission of the file transmission and before the file transmission is complete, without requiring that the entire sequence of received packets be subjected to n-gram analysis.