Systems and methods of malware detection
Systems and methods for detecting suspicious malware by analyzing data such as transfer protocol data or logs from a host within an enterprise is provided. The systems and methods include a database for storing current data and historical data obtained from the network and a detection module and an optional display. The embodiments herein extract information from non-encrypted transfer protocol metadata, determine a plurality of features, utilize an outlier detection model that is based on historical behaviors, calculate a suspiciousness score, and create alerts for analysis by users when the score exceeds a threshold. In doing so, the systems and methods of the present invention improve the ability to identify suspicious outliers or potential malware on an iterative basis over time.
1. A system for detecting malicious traffic in a network comprising:
at least one sensor, wherein each sensor is configured to:
mirror network traffic for at least one device connected to the network; and
create a set of data consisting of transfer protocol records and associated transfer protocol record metadata by, over a time interval, parsing the mirrored network traffic for transfer protocol records;
a database coupled to the at least one sensor and configured to:
receive and store the set of data from each of the at least one sensor, wherein the data received and stored during the time interval comprise a set of test data;
store a set of historical data previously obtained from the network, wherein the historical data consists of transfer protocol records and associated transfer protocol metadata; and
a computation engine including at least one processor and non-transitory memory, the computation engine including a detection module configured to run on the computation engine and adapted and configured to perform the steps of:
loading the set of test data from the database;
filtering the set of test data to obtain filtered data based on at least one criterion,
wherein the filtered data includes data for at least one transfer protocol record;
saving the filtered data to the database;
determining a value of each of a plurality of features of each transfer protocol record;
loading, for each transfer protocol record, a set of previously computed historical feature values from the database, wherein the historical feature values were computed from the set of historical data; and
computing an output score for each transfer protocol record based on comparing, for each feature of each transfer protocol record, the determined value to the set of previously computed historical feature values for that feature, wherein the output score for each transfer protocol record indicates the likelihood that that transfer protocol record represents malicious network traffic.
2. The system of claim 1 , wherein the at least one criterion is one of a file path information, file name, a content type, a content length, and a file extension type.
3. The system of claim 1 , wherein the transfer protocol is one of hypertext transfer protocol (HTTP), file transfer protocol (FTP), server message block (SMB), and simple mail transfer protocol (SMTP).
4. The system of claim 1 , wherein the detection module is further adapted and configured to perform the step of creating an alert for each output score at or above a predetermined threshold.
5. The system of claim 4 , further comprising a display for displaying the alert received from the detection module.
6. The system of claim 1 , wherein the plurality of features includes at least one of: a count of a number of times downloads are made from an observed protocol host over a time interval, a count of a number of times an observed transfer protocol path is downloaded over a time interval, an amount by which the value of one feature within the plurality of features is abnormal relative to other file downloads with a same extension as the one feature, and a determination of how strongly a downloaded file name correlates with a list of known malware file names.
7. The system of claim 1 , wherein the set of historical data consists of data received from the network during a predetermined period of time preceding the time interval.
8. The system of claim 1 , wherein the at least one criterion for filtering includes removing each transfer protocol record that is empty or does not include a file extension.
9. The system of claim 1 , wherein the set of historical data is stored over a predetermined period of time having a length and a temporal distance prior to the time interval.
10. The system of claim 1 , wherein the comparing the determined value to the set of previously computed historical feature values for that feature further comprises calculating a Z-score or a P-score for each feature.
11. A method for detecting malicious traffic in a network comprising the steps of:
mirroring network traffic for at least one device connected to the network by each of at least one sensor connected to the network;
creating, by each of the at least one sensor, a set of data consisting of transfer protocol records and associated transfer protocol record metadata by, over a time interval, parsing the mirrored network traffic for transfer protocol records;
receiving and storing, by a database coupled to the at least one sensor, the set of data from each of the at least one sensor, wherein the data received and stored during the time interval comprise a set of test data;
storing, by the database, a set of historical data previously obtained from the network, wherein the historical data consists of transfer protocol records and associated transfer protocol metadata;
loading the set of test data from the database;
filtering the set of test data to obtain filtered data based on at least one criterion, wherein the filtered data includes data for at least one transfer protocol record;
saving the filtered data to the database;
determining a value of each of a plurality of features of the filtered data;
loading, for each of the plurality of features of the filtered data, a set of previously computed historical feature values from the database, wherein the historical feature values were computed from the set of historical data;
determining a value of each of a plurality of features of each transfer protocol record;
loading, for each transfer protocol record, a set of previously computed historical feature values from the database, wherein the historical feature values were computed from the set of historical data; and
computing an output score for each transfer protocol record based on comparing, for each feature of each transfer protocol record, the determined value to the set of previously computed historical feature values for that feature, wherein the output score for each transfer protocol record indicates the likelihood that that transfer protocol record represents malicious network traffic.
12. The method of claim 11 , wherein the at least one criterion is one of a file path information, a file name, a content type, a content length, and a file extension type.
13. The method of claim 11 , wherein the transfer protocol is one of hypertext transfer protocol (HTTP), file transfer protocol (FTP), server message block (SMB), and simple mail transfer protocol (SMTP).
14. The method of claim 11 , further comprising the step of creating an alert for each output score at or above a predetermined threshold.
15. The method of claim 14 , further comprising displaying the alert in a display.
16. The method of claim 11 , wherein the plurality of features includes at least one of: a count of a number of times downloads are made from an observed protocol host over a time interval, a count of a number of times an observed transfer protocol path is downloaded over a time interval, an amount by which the value of one feature within the plurality of features is abnormal relative to other file downloads with a same extension, and a determination of how strongly a downloaded file name correlates with a list of known malware file names.
17. The method of claim 11 , wherein the set of historical data consists of data received from the network during a predetermined period of time preceding the time interval.
18. The method of claim 11 , wherein the set of historical data is stored over a predetermined period of time having a length and a temporal distance prior to the time interval.
19. The method of claim 11 , wherein the comparing the determined value to the set of previously computed historical feature values for that feature further comprises calculating a Z-score or a P-score for each feature.