Methods and systems for analyzing cybersecurity threats
To analyze cybersecurity threats, an analysis module of a processor may receive log data from at least one network node. The analysis module may identify at least one statistical outlier within the log data. The analysis module may determine that the at least one statistical outlier represents a cybersecurity threat by applying at least one machine learning algorithm to the at least one statistical outlier.
1. A method of analyzing cybersecurity threats comprising:
performing processing associated with receiving, with an analysis module of a processor, log data from a network node;
performing processing associated with identifying, with the analysis module, a statistical outlier within the log data using Tailjumps, the identifying comprising:
transforming the log data into community data indicative of a network structure, identifying a plurality of communications between nodes in the community data, and identifying the statistical outlier within the plurality of communications;
generating a behavioral graph model using the log data, determining a best fit line for a characteristic of the behavioral graph model, and identifying a data point at or beyond a predetermined distance from the best fit line as the statistical outlier; or
analyzing text data in the log data using language processing algorithm to identify the statistical outlier; or
a combination thereof; and
performing processing associated with determining, with the analysis module, that the statistical outlier represents a cybersecurity threat by applying a Janus machine learning algorithm to provide a predication about the novelty of the statistical outlier,
wherein the Janus machine learning algorithm categorizes the statistical outlier into classes, wherein the classes are separated by a security threat decision boundary, and wherein the classes comprise: a forward class that is a most probable security threat class, a do not send class that is a least probable security threat class, and an unknown class for entities that are on the security threat decision boundary;
performing processing associated with forwarding statistical outliers categorized as unknown for additional analysis;
performing processing associated with updating the security threat decision boundary based on analysist input; and
performing processing associated with repeating the categorizing for the statistical outliers categorized as unknown with an updated security threat decision boundary.
2. The method of claim 1 , wherein the processing associated with transforming the log data into community data comprises determining connections between network nodes from the log data.
3. The method of claim 1 , wherein the processing associated with analyzing the text data comprises generating a feature, modeling the feature using an autoencoder to learn the underlying structure, and finding a deviation from the underlying structure.
4. The method of claim 3 , wherein the feature is generated using Word2Vec.
5. The method of claim 3 , wherein the modeling is performed using a neural network.
6. The method of claim 1 , wherein the processing associated with identifying the statistical outlier comprises filtering the community data, the behavioral graph model, or the text data, or a combination thereof to ignore all but extreme values within the community data, the behavioral graph model, or the text data, or the combination thereof.
7. The method of claim 1 , wherein the processing associated with identifying the statistical outlier comprises filtering the community data, the behavioral graph model, or the text data, or a combination thereof to select data within the community data, the behavioral graph model, or the text data, or the combination thereof that lies within a determined region of interest.
8. The method of claim 7 , wherein the determined region of interest comprises a region outside a clear inlier region.
9. The method of claim 7 , wherein the determined region of interest comprises a forecast region.
10. The method of claim 7 , further comprising performing processing associated with analyzing, with the analysis module, the community data, the behavioral graph model, or the text data, or the combination thereof to determine the determined region of interest.
11. The method of claim 1 , wherein the processing associated with determining that the statistical outlier represents a cybersecurity threat comprises determining whether the statistical outlier has a feature from a suspicious feature set.
12. The method of claim 11 , wherein the suspicious feature set is generated by training a machine learning algorithm.
13. The method of claim 12 , further comprising performing processing associated with training, with the analysis module, the machine learning algorithm.
14. The method of claim 13 , wherein the training comprises applying past ground-truth results generated by the analysis module to the suspicious feature set.
15. The method of claim 13 , wherein the training comprises receiving a user input.
16. The method of claim 1 , wherein the processing associated with determining that the statistical outlier represents a cybersecurity threat comprises determining that the statistical outlier is a periodic behavior.
17. The method of claim 1 , wherein the machine learning algorithm comprises a random forest algorithm for self-training.
18. The method of claim 17 , wherein the random forest includes an unknown class for active learning.
19. The method of claim 1 , further comprising performing processing associated with analyzing a domain generation algorithm (DGA) using a generic probabilistic model.
20. The method of claim 19 , wherein the processing associated with analyzing the DGA comprises aggregating requests generated by the DGA, discarding domains requested by the DGA, or analyzing similarity of requests generated by the DGA, or a combination thereof.
21. The method of claim 1 , further comprising performing processing associated with analyzing a user-generated request by applying a user agent anomaly detection algorithm.
22. The method of claim 21 , further comprising performing processing associated with training the user agent machine learning algorithm.
23. A system for analyzing cybersecurity threats comprising:
a processor;
a network interface in communication with the processor; and
an analysis module of the processor configured to:
perform processing associated with receiving log data from network node by the network interface;
perform processing associated with identifying statistical outlier using Tailjumps within the log data, the identifying comprising:
transforming the log data into community data indicative of a network structure, identifying a plurality of communications between nodes in the community data, and identifying the statistical outlier within the plurality of communications;
generating a behavioral graph model using the log data, determining a best fit line for characteristic of the behavioral graph model, and identifying data point at or beyond a predetermined distance from the best fit line as the statistical outlier; or
analyzing text data in the log data using language processing algorithm to identify the statistical outlier; or
a combination thereof; and
perform processing associated with determining that the statistical outlier represents a cybersecurity threat by applying a Janus machine learning algorithm to provide a predication about the novelty of the statistical outlier,
wherein the Janus machine learning algorithm categorizes the statistical outlier into classes, wherein the classes are separated by a security threat decision boundary, and wherein the classes comprise: a forward class that is a most probable security threat class, a do not send class that is a least probable security threat class, and an unknown class for entities that are on the security threat decision boundary;
performing processing associated with forwarding statistical outliers categorized as unknown for additional analysis;
performing processing associated with updating the security threat decision boundary based on analysist input; and
performing processing associated with repeating the categorizing for the statistical outliers categorized as unknown with an updated security threat decision boundary.
24. The system of claim 23 , wherein the processing associated with transforming the log data into community data comprises determining connections between network nodes from the log data.
25. The system of claim 23 , wherein the processing associated with analyzing the text data comprises generating a feature, modeling the feature using an autoencoder to learn the underlying structure, and finding a deviation from the underlying structure.
26. The system of claim 25 , wherein the feature is generated using Word2Vec.
27. The system of claim 25 , wherein the modeling is performed using a neural network.
28. The system of claim 23 , wherein the processing associated with identifying the statistical outlier comprises filtering the community data, the behavioral graph model, or the text data, or a combination thereof to ignore all but extreme values within the community data, the behavioral graph model, or the text data, or the combination thereof.
29. The system of claim 23 , wherein the processing associated with identifying the statistical outlier comprises filtering the community data, the behavioral graph model, or the text data, or a combination thereof to select data within the community data, the behavioral graph model, or the text data, or the combination thereof that lies within determined region of interest.
30. The system of claim 29 , wherein the determined region of interest comprises a region outside a clear inlier region.
31. The system of claim 29 , wherein the determined region of interest comprises a forecast region.
32. The system of claim 29 , wherein the analysis module is further configured to perform processing associated with analyzing the community data, the behavioral graph model, or the text data, or the combination thereof to determine the determined region of interest.
33. The system of claim 23 , wherein the processing associated with determining that the statistical outlier represents a cybersecurity threat comprises determining whether the statistical outlier has a feature from a suspicious feature set.
34. The system of claim 33 , wherein the suspicious feature set is generated by training the machine learning algorithm.
35. The system of claim 34 , wherein the analysis module is further configured to perform processing associated with training the machine learning algorithm.
36. The system of claim 35 , wherein the training comprises applying past ground-truth results generated by the analysis module to the suspicious feature set.
37. The system of claim 35 , wherein the training comprises receiving a user input.
38. The system of claim 23 , wherein the processing associated with determining that the statistical outlier represents a cybersecurity threat comprises determining that the statistical outlier is a periodic behavior.
39. The system of claim 23 , wherein the machine learning algorithm comprises a random forest algorithm for self-training.
40. The system of claim 39 , wherein the random forest includes an unknown class for active learning.
41. The system of claim 23 , wherein the analysis module is further configured to perform processing associated with analyzing a domain generation algorithm (DGA) using a generic probabilistic model.
42. The system of claim 41 , wherein the processing associated with analyzing the at least fone DGA comprises aggregating requests generated by the DGA, discarding domains requested by the DGA, or analyzing similarity of requests generated by the DGA, or a combination thereof.
43. The system of claim 23 , wherein the analysis module is further configured to perform processing associated with analyzing a user-generated request by applying a user agent machine learning algorithm.
44. The system of claim 43 , further comprising performing processing associated with training the user agent machine learning algorithm.