Machine learning detection of network attacks using traffic and log information

1. An Intrusion Detection System (IDS) comprising:

a processing device, and

a memory device configured to store a computer program having instructions that, when executed, enable the processing device to

obtain network traffic information regarding data traffic in a network under observation,

obtain system log information regarding operations of the network under observation,

insert the network traffic information and the system log information into multiple different machine learning analysis procedures, wherein each machine learning analysis procedure of the multiple different machine learning analysis procedures is configured to detect a different sub-attack of a multi-stage attack to which the network under observation is susceptible,

combine the outputs of the multiple different machine learning analysis procedures to detect whether and determine a time period during which one or more sub-attacks have been launched against the network under observation, and

in response to detecting that one or more sub-attacks have been launched during the determined time period, determine the type of the one or more sub-attacks.

2. The IDS of claim 1 , wherein each of the multiple different machine learning analysis procedures uses a machine learning model.

3. The IDS of claim 2 , wherein each of the machine learning models includes one or more of a Deep Neural Network (DNN) technique, an ensemble technique, a nearest neighbor technique, and a clustering technique.

4. The IDS of claim 2 , wherein the instructions further enable the processing device to use a word-embedding procedure when one or more of the network traffic information and system log information is in textual form, and wherein the word-embedding procedure is configured to change a textual representation of the information into a numerical representation used by the machine learning model.

5. The IDS of claim 2 , wherein each of the machine learning models is configured to predict a sub-attack based on F1 scores.

6. The IDS of claim 1 , wherein the one or more sub-attacks include at least reconnaissance, initial attack, pivoting, lateral movement, and data exfiltration.

7. The IDS of claim 1 , wherein the one or more sub-attacks include at least reconnaissance, resource development, initial access, execution, persistence, privilege escalation, defense evasion, credential access, discovery, lateral movement, collection, command and control, exfiltration, and impact.

8. The IDS of claim 1 , wherein the system log information includes one or more of server logs, router logs, network element logs, equipment logs, device logs, event logs, transaction logs, and message logs.

9. The IDS of claim 1 , wherein the network traffic information and system log information are obtained in a time period having a length that is below a predetermined threshold.

10. The IDS of claim 1 , wherein the network traffic information is detected using one or more of NetFlow, sFlow, IPFIX, and PCAP.

11. The IDS of claim 1 , wherein obtaining the network traffic information and system log information includes receiving the network traffic information and system log information from an edge element at an edge of the network under observation.

12. The IDS of claim 1 , wherein the IDS includes combined features of a Network Intrusion Detection System (NIDS) and a Host-based Intrusion Detection System (HIDS).

13. A non-transitory computer-readable medium configured to store computer logic having instructions that, when executed, enable a processing device to:

obtain network traffic information regarding data traffic in a network under observation,

obtain system log information regarding operations of the network under observation,

insert the network traffic information and the system log information into multiple different machine learning analysis procedures, wherein each machine learning analysis procedure of the multiple different machine learning analysis procedures is configured to detect a different sub-attack of a multi-stage attack to which the network under observation is susceptible,

combine the outputs of the multiple different machine learning analysis procedures to detect whether and determine a time period during which one or more sub-attacks have been launched against the network under observation, and

in response to detecting that one or more sub-attacks have been launched during the determined time period, determine the type of the one or more sub-attacks.

14. The non-transitory computer-readable medium of claim 13 , wherein each of the multiple different machine learning analysis procedures uses a machine learning model.

15. The non-transitory computer-readable medium of claim 14 , wherein each of the machine learning models includes one or more of a Deep Neural Network (DNN) technique, an ensemble technique, a nearest neighbor technique, and a clustering technique.

16. The non-transitory computer-readable medium of claim 14 , wherein the instructions further enable the processing device to use a word-embedding procedure when one or more of the network traffic information and system log information is in textual form, and wherein the word-embedding procedure is configured to change a textual representation of the information into a numerical representation used by the machine learning model.

17. A method comprising the steps of:

obtaining network traffic information regarding data traffic in a network under observation,

obtaining system log information regarding operations of the network under observation,

inserting the network traffic information and the system log information into multiple different machine learning analysis procedures, wherein each machine learning analysis procedure of the multiple different machine learning analysis procedures is configured to detect a respective sub-attack of a multi-stage attack to which the network under observation is susceptible,

combining the outputs of the multiple different machine learning analysis procedures to detect whether and determine a time period during which one or more sub-attacks have been launched against the network under observation, and

in response to detecting that one or more sub-attacks have been launched during the determined time period, determining the type of the one or more sub-attacks.

18. The method of claim 17 , wherein the one or more sub-attacks include at least reconnaissance, resource development, initial access, execution, persistence, privilege escalation, defense evasion, credential access, discovery, lateral movement, collection, command and control, exfiltration, and impact, and wherein the system log information includes one or more of server logs, router logs, network element logs, equipment logs, device logs, event logs, transaction logs, and message logs.

19. The method of claim 17 , wherein the network traffic information and system log information are obtained from an edge element at an edge of the network under observation in a time period having a length that is below a predetermined threshold.

20. The method of claim 17 , wherein the network traffic information is detected using one or more of NetFlow, sFlow, IPFIX, and PCAP.