Machine learning based botnet detection using real-time extracted traffic features

1. A method for identifying a botnet in a network, comprising:

obtaining historical network data in the network, the historical network data comprising a first plurality of data units;

obtaining a ground truth data set associated with the historical network data, the ground truth data set comprising a plurality of labels with each label assigned to a corresponding data unit of the first plurality of data units, said each label comprising one of a first label categorizing said corresponding data unit as associated with the botnet and a second label categorizing said corresponding data unit as not associated with the botnet;

analyzing, by a central processing unit (CPU) of a computer and using a machine learning algorithm, the historical network data and the ground truth data set to generate a model comprising a decision tree and a plurality of functions each associated with one of a plurality of leaf nodes of the decision tree, wherein the decision tree comprises the plurality of leaf nodes, a first non-leaf node, and a second non-leaf node downstream from the first non-leaf node, wherein the historical network data is applied as input to the decision tree and is split while traversing the decision tree, wherein the first non-leaf node splits a first portion of the historical network data to traverse two paths of the decision tree, wherein the first portion of the historical network data is split based on a packet inter-arrival time statistics of the first plurality of data units, wherein the second non-leaf node splits a second portion of the historical network data traversing one of the two paths of the decision tree to the second non-leaf node, wherein the second portion of the historical network data is split based on at least one selected from a group consisting of a layer-4 payload size statistics and a number of bytes per packet statistics of the first plurality of data units, wherein each of the plurality of functions outputs a first statistical prediction of the plurality of labels assigned to a third portion of the historical network data traversing the decision tree to a corresponding leaf node of the plurality of leaf nodes, wherein the machine learning algorithm adjusts the model to match the first statistical prediction to the ground truth data set;

obtaining real-time network data in the network, the real-time network data comprising a second plurality of data units;

applying the second plurality of data units as the input to the decision tree to output a second statistical prediction of the plurality of labels assigned to a second data unit of the second plurality of data units; and

categorizing the second data unit as associated with the botnet based on the second statistical prediction.

2. The method of claim 1 , wherein the first plurality of data units comprise a plurality of flow tuples, each tuple comprising a source IP address, a destination IP address, a destination port, and a protocol identifier.

3. The method of claim 2 ,

wherein the packet inter-arrival time statistics, the layer-4 payload size statistics, and the number of bytes per packet statistics are time dependent and comprise a first set of values corresponding to a first version of the historical network data within a first time window and a second set of values corresponding to a second version of the historical network data within a second time window,

wherein analyzing the historical network data comprises:

representing the first and second sets of values as first and second matrices; and

generating a time averaged matrix from the first and second matrices based on a pre-determined statistical formula,

wherein the historical network data is split based on the time averaged matrix while traversing the decision tree.

4. The method of claim 3 , wherein the pre-determined statistical formula comprises exponentially weighted moving average (EWMA).

5. The method of claim 1 , wherein the machine learning algorithm comprises Multi-layer perceptron.

6. A system for identifying a botnet in a network, comprising:

a processor; and

memory storing instructions, when executed by the processor, comprising functionalities for:

obtaining historical network data in the network, the historical network data comprising a first plurality of data units;

obtaining a ground truth data set associated with the historical network data, the ground truth data set comprising a plurality of labels with each label assigned to a corresponding data unit of the first plurality of data units, said each label comprising one of a first label categorizing said corresponding data unit as associated with the botnet and a second label categorizing said corresponding data unit as being not associated with the botnet;

analyzing, using a machine learning algorithm, the historical network data and the ground truth data set to generate a model comprising a decision tree and a plurality of functions each associated with one of a plurality of leaf nodes of the decision tree, wherein the decision tree comprises the plurality of leaf nodes, a first non-leaf node, and a second non-leaf node downstream from the first non-leaf node, wherein the historical network data is applied as input to the decision tree and is split while traversing the decision tree, wherein the first non-leaf node splits a first portion of the historical network data to traverse two paths of the decision tree, wherein the first portion of the historical network data is split based on a packet inter-arrival time statistics of the first plurality of data units, wherein the second non-leaf node splits a second portion of the historical network data traversing one of the two paths of the decision tree to the second non-leaf node, wherein the second portion of the historical network data is split based on at least one selected from a group consisting of a layer-4 payload size statistics and a number of bytes per packet statistics of the first plurality of data units, wherein each of the plurality of functions outputs a first statistical prediction of the plurality of labels assigned to a third portion of the historical network data traversing the decision tree to a corresponding leaf node of the plurality of leaf nodes, wherein the machine learning algorithm adjusts the model to match the first statistical prediction to the ground truth data set;

obtaining real-time network data in the network, the real-time network data comprising a second plurality of data units;

applying the second plurality of data units as the input to the decision tree to output a second statistical prediction of the plurality of labels assigned to a second data unit of the second plurality of data units; and

categorizing the second data unit as associated with the botnet based on the second statistical prediction.

7. The system of claim 6 , wherein the first plurality of data units comprise a plurality of flow tuples, each tuple comprising a source IP address, a destination IP address, a destination port, and a protocol identifier.

8. The system of claim 7 ,

wherein the packet inter-arrival time statistics, the layer-4 payload size statistics, and the number of bytes per packet statistics are time dependent and comprise a first set of values corresponding to a first version of the historical network data within a first time window and a second set of values corresponding to a second version of the historical network data within a second time window,

wherein analyzing the historical network data comprises:

representing the first and second sets of values as first and second matrices; and

generating a time averaged matrix from the first and second matrices based on a pre-determined statistical formula,

wherein the historical network data is split based on the time averaged matrix while traversing the decision tree.

9. The system of claim 8 , wherein the pre-determined statistical formula comprises exponentially weighted moving average (EWMA).

10. The system of claim 6 , wherein the machine learning algorithm comprises Multi-layer perceptron.

11. A non-transitory computer readable medium storing instructions for identifying a botnet in a network, the instructions, when executed by a processor of a computer, comprising functionality for:

obtaining historical network data in the network, the historical network data comprising a first plurality of data units;

obtaining a ground truth data set associated with the historical network data, the ground truth data set comprising a plurality of labels with each label assigned to a corresponding data unit of the first plurality of data units, said each label comprising one of a first label categorizing said corresponding data unit as associated with the botnet and a second label categorizing said corresponding data unit as not associated with the botnet;

analyzing, using a machine learning algorithm, the historical network data and the ground truth data set to generate a model comprising a decision tree and a plurality of functions each associated with one of a plurality of leaf nodes of the decision tree, wherein the decision tree comprises the plurality of leaf nodes, a first non-leaf node, and a second non-leaf node downstream from the first non-leaf node, wherein the historical network data is applied as input to the decision tree and is split while traversing the decision tree, wherein the first non-leaf node splits a first portion of the historical network data to traverse two paths of the decision tree, wherein the first portion of the historical network data is split based on a packet inter-arrival time statistics of the first plurality of data units, wherein the second non-leaf node splits a second portion of the historical network data traversing one of the two paths of the decision tree to the second non-leaf node, wherein the second portion of the historical network data is split based on at least one selected from a group consisting of a layer-4 payload size statistics and a number of bytes per packet statistics of the first plurality of data units, wherein each of the plurality of functions outputs a first statistical prediction of the plurality of labels assigned to a third portion of the historical network data traversing the decision tree to a corresponding leaf node of the plurality of leaf nodes, wherein the machine learning algorithm adjusts the model to match the first statistical prediction to the ground truth data set;

obtaining real-time network data in the network, the real-time network data comprising a second plurality of data units;

applying the second plurality of data units as the input to the decision tree to output a second statistical prediction of the plurality of labels assigned to a second data unit of the second plurality of data units; and

categorizing the second data unit as associated with the botnet based on the second statistical prediction.

12. The non-transitory computer readable medium of claim 11 , wherein the first plurality of data units comprise a plurality of flow tuples, each tuple comprising a source IP address, a destination IP address, a destination port, and a protocol identifier.

13. The non-transitory computer readable medium of claim 12 ,

wherein the packet inter-arrival time statistics, the layer-4 payload size statistics, and the number of bytes per packet statistics are time dependent and comprise a first set of values corresponding to a first version of the historical network data within a first time window and a second set of values corresponding to a second version of the historical network data within a second time window,

wherein analyzing the historical network data comprises:

representing the first and second sets of values as first and second matrices; and

generating a time averaged matrix from the first and second matrices based on a pre-determined statistical formula,

wherein the historical network data is split based on the time averaged matrix while traversing the decision tree.

14. The non-transitory computer readable medium of claim 13 , wherein the pre-determined statistical formula comprises exponentially weighted moving average (EWMA).

15. The non-transitory computer readable medium of claim 11 , wherein the machine learning algorithm comprises Multi-layer perceptron.