Anomaly detection with K-means clustering and artificial outlier injection
The present disclosure is directed towards systems and methods for improving anomaly detection using injected outliers. A normalcy calculator of a device may include a set of outliers into a training dataset of data points. The normalcy calculator, using a K-means clustering algorithm applied on the training dataset, identify at least a first cluster of data points. The normalcy calculator of the device may determine a region with a center and an outer radius that covers at least a spatial extent of the first cluster of data points. The normalcy calculator may determine a first normalcy radius for the first cluster by reducing the region around the center until a point at which all artificial outliers are excluded from a region defined by the first normalcy radius. An outlier detector of the device may use the region defined by the first normalcy radius to determine whether a new data point is normal or abnormal.
1. A method for improving anomaly detection using injected outliers, the method comprising:
including, by a device, a set of outliers into a training dataset of data points;
identifying, by the device using a K-means clustering algorithm applied on the training dataset, at least a first cluster of data points;
determining, by a normalcy calculator of the device, a region with a center and an outer radius that covers at least a spatial extent of the first cluster of data points;
determining, by the normalcy calculator, a first normalcy radius for the first cluster by adjusting the region around the center until a point at which all outliers are excluded from the adjusted region and the adjusted region is defined by the first normalcy radius; and
using, by an outlier detector of the device, the region defined by the first normalcy radius to determine whether a new data point is normal or abnormal.
2. The method of claim 1 , wherein the training dataset of data points comprises a dataset of network traffic data points, and
wherein the new data point comprises a new network traffic data point of network traffic; and further comprising:
determining, by the outlier detector, whether the network traffic has anomalous behavior based on the determination that the new network traffic data point is normal or abnormal using the region defined by the first normalcy radius.
3. The method of claim 1 , wherein identifying at least the first cluster of data points comprises identifying at least the first cluster of data points using the K-means clustering algorithm in semi-supervised mode.
4. The method of claim 1 , comprising determining that the new data point is normal responsive to determining that the new data point reside within the region defined by the first normalcy radius.
5. The method of claim 1 , comprising determining that the new data point is abnormal responsive to at least determining that the new data point reside beyond the region defined by the first normalcy radius.
6. The method of claim 1 , further comprising identifying the first cluster of data points and a second cluster of data points using the K-means clustering algorithm.
7. The method of claim 6 , further comprising determining a second normalcy radius for the second cluster, and using a region defined by the second normalcy radius to determine whether the new data point is normal or abnormal.
8. The method of claim 7 , determining that the new data point is abnormal responsive to at least determining that the new data point reside beyond the region defined by the first normalcy radius and the region defined by the second normalcy radius.
9. The method of claim 1 , comprising:
receiving, by the device, a network traffic, the network traffic corresponding to the new data point, the device being intermediary between at least one client and at least one server; and
determining, by the device using the outlier detector, if the network traffic is normal or abnormal.
10. The method of claim 1 , further comprising reducing a number of data points incorrectly determined to be normal, using the region defined by the first normalcy radius.
11. A system for improving anomaly detection using injected outliers, the system comprising:
a device configured to perform anomaly detection;
a normalcy calculator of a device, the normalcy calculator configured to:
include a set of outliers into a training dataset of data points;
identify, using a K-means clustering algorithm applied on the training dataset, at least a first cluster of data points;
determine a center and an outer radius of a region that covers at least a spatial extent of the first cluster of data points; and
determine a first normalcy radius for the first cluster by adjusting the region around the center until a point at which all outliers are excluded from the adjusted region and the adjusted region is defined by the first normalcy radius; and
an outlier detector of the device, the outlier detector configured to use the region defined by the first normalcy radius to determine whether a new data point is normal or abnormal.
12. The system of claim 11 , wherein the training dataset of data points comprises a dataset of network traffic data points,
wherein the new data point comprises a new network traffic data point of network traffic; and
wherein the outlier detector is further configured to determine whether the network traffic has anomalous behavior based on the determination that the new network traffic data point is normal or abnormal using the region defined by the first normalcy radius.
13. The system of claim 11 , wherein the normalcy calculator is configured to identify the at least the first cluster of data points using the K-means clustering algorithm in semi-supervised mode.
14. The system of claim 11 , wherein the outlier detector is configured to determine that the new data point is normal responsive to determining that the new data point reside within the region defined by the first normalcy radius.
15. The system of claim 11 , wherein the outlier detector is configured to determine that the new data point is abnormal responsive to at least determining that the new data point reside beyond the region defined by the first normalcy radius.
16. The system of claim 11 , wherein the normalcy calculator is configured to identify the first cluster of data points and a second cluster of data points using the K-means clustering algorithm.
17. The system of claim 16 , wherein the normalcy calculator is configured to determine a second normalcy radius for the second cluster, and the outlier detector is configured to use a region defined by the second normalcy radius to determine whether the new data point is normal or abnormal.
18. The system of claim 17 , wherein the outlier detector is configured to determine that the new data point is abnormal responsive to at least determining that the new data point reside beyond the region defined by the first normalcy radius and the region defined by the second normalcy radius.
19. The system of claim 11 , wherein the device is intermediary between at least one client and at least one server, the device is configured to:
receive a network traffic, the network traffic corresponding to the new data point; and determine, using the outlier detector, if the network traffic is normal or abnormal.
20. The system of claim 11 , wherein the normalcy calculator is configured to reduce a number of data points incorrectly determined to be normal, using the region defined by the first normalcy radius.