Anomaly detection and mitigation using device subpopulation partitioning
A system and method for anomaly detection. A method includes recursively partitioning a sample of device activity data including deterministic characteristics of a population of devices over iterations in order to create partitions. Each iteration includes determining a split density metric for a candidate subpopulation created by splitting a portion of the population with respect to a corresponding type of deterministic characteristic. The split density metric for the candidate subpopulation is determined based on a density value of the candidate subpopulation and a coverage value of the corresponding type of deterministic characteristic. The partitions include each candidate subpopulation meeting a split density metric threshold. A baseline for each of the partitions is established based on device activity for devices represented in device activity data of the partition. An anomaly is detected based on behavior of a device and the baseline established for a partition corresponding to the device.
1. A method for anomaly detection using device subpopulation partitioning,
comprising:
recursively partitioning a sample of device activity data over a plurality of iterations in order to create a plurality of partitions, wherein the device activity data includes deterministic characteristics of a population including a plurality of devices, wherein each iteration of recursively partitioning the sample further comprises:
determining a split density metric for a candidate subpopulation of a plurality of subpopulations, wherein each of the plurality of subpopulations is created by splitting at least a portion of the population with respect to a corresponding type of deterministic characteristic, wherein the split density metric for the candidate subpopulation is determined based on a density value of the candidate subpopulation and a coverage value of the corresponding type of deterministic characteristic; and
checking whether the candidate subpopulation meets a split density metric threshold by comparing the split density metric for the candidate subpopulation to the split density metric threshold, wherein the plurality of partitions includes each candidate subpopulation that meets the split density metric threshold;
establishing a baseline for each of the plurality of partitions, wherein the baseline established for each partition is determined based on device activity for devices represented in a portion of the device activity data of the partition; and
detecting an anomaly based on behavior of a device and the baseline established for a partition corresponding to the device.
2. The method of claim 1 , further comprising:
identifying at least one failed subpopulation of the plurality of subpopulations having a split density metric which fails the split density metric threshold; and
creating at least one clustering partition by applying a clustering model to at least one portion of the device activity data corresponding to devices of the at least one failed subpopulation, wherein the plurality of partitions includes the at least one clustering partition.
3. The method of claim 2 , wherein the clustering model is a Gaussian mixture model.
4. The method of claim 1 , wherein the candidate subpopulation at each iteration is selected from among a plurality of candidate subpopulations for the iteration, wherein the plurality of candidate subpopulations for each iteration represents a plurality of distinct values for the corresponding type of deterministic characteristic of the iteration.
5. The method of claim 1 , further comprising:
identifying at least one failed subpopulation of the plurality of subpopulations for which all potential types of deterministic characteristics have been exhausted, wherein the potential types of deterministic characteristics are a predetermined set of types of characteristics to be used for partitioning the population, wherein all potential types of deterministic characteristics have been exhausted for a failed subpopulation of the plurality of subpopulations when the failed subpopulation was created by splitting the device activity data once using each of the potential types of deterministic characteristics; and
creating at least one clustering partition by applying a clustering model to at least one portion of the device activity data corresponding to devices of the at least one failed subpopulation, wherein the plurality of partitions includes the at least one clustering partition.
6. The method of claim 1 , wherein determining the split density metric for the candidate subpopulation at each iteration further comprises:
generating a distance matrix for the candidate subpopulation, wherein the density value is determined based on the distance matrix.
7. The method of claim 6 , wherein the distance matrix includes a plurality of rows, wherein determining the split density metric for the candidate subpopulation at each iteration further comprises:
determining a plurality of median distance values for the plurality of rows, wherein the density value is determined based on the plurality of median distance values.
8. The method of claim 7 , wherein the density value determined for the candidate subpopulation at each iteration is determined as a median of the plurality of median distance values.
9. The method of claim 1 , further comprising:
performing at least one mitigation action with respect to the detected anomaly.
10. A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to execute a process, the process comprising:
recursively partitioning a sample of device activity data over a plurality of iterations in order to create a plurality of partitions, wherein the device activity data includes deterministic characteristics of a population including a plurality of devices, wherein each iteration of recursively partitioning the sample further comprises:
determining a split density metric for a candidate subpopulation of a plurality of subpopulations, wherein each of the plurality of subpopulations is created by splitting at least a portion of the population with respect to a corresponding type of deterministic characteristic, wherein the split density metric for the candidate subpopulation is determined based on a density value of the candidate subpopulation and a coverage value of the corresponding type of deterministic characteristic; and
checking whether the candidate subpopulation meets a split density metric threshold by comparing the split density metric for the candidate subpopulation to the split density metric threshold, wherein the plurality of partitions includes each candidate subpopulation that meets the split density metric threshold;
establishing a baseline for each of the plurality of partitions, wherein the baseline established for each partition is determined based on device activity for device represented in a portion of the device activity data of the partition; and
detecting an anomaly based on behavior of a device and the baseline established for a partition corresponding to the device.
11. A system for anomaly detection using device subpopulation partitioning, comprising:
a processing circuitry; and
a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to:
recursively partition a sample of device activity data over a plurality of iterations in order to create a plurality of partitions, wherein the device activity data includes deterministic characteristics of a population including a plurality of devices, wherein the system is further configured to, at each iteration:
determine a split density metric for a candidate subpopulation of a plurality of subpopulations, wherein each of the plurality of subpopulations is created by splitting at least a portion of the population with respect to a corresponding type of deterministic characteristic, wherein the split density metric for the candidate subpopulation is determined based on a density value of the candidate subpopulation and a coverage value of the corresponding type of deterministic characteristic; and
check whether the candidate subpopulation meets a split density metric threshold by comparing the split density metric for the candidate subpopulation to the split density metric threshold, wherein the plurality of partitions includes each candidate subpopulation that meets the split density metric threshold;
establish a baseline for each of the plurality of partitions, wherein the baseline established for each partition is determined based on device activity for devices represented in a portion of the device activity data of the partition; and
detect an anomaly based on behavior of a device and the baseline established for a partition corresponding to the device.
12. The system of claim 11 , wherein the system is further configured to:
identify at least one failed subpopulation of the plurality of subpopulations having a split density metric which fails the split density metric threshold; and
create at least one clustering partition by applying a clustering model to at least one portion of the device activity data corresponding to devices of the at least one failed subpopulation, wherein the plurality of partitions includes the at least one clustering partition.
13. The system of claim 12 , wherein the clustering model is a Gaussian mixture model.
14. The system of claim 11 , wherein the candidate subpopulation at each iteration is selected from among a plurality of candidate subpopulations for the iteration, wherein the plurality of candidate subpopulations for each iteration represents a plurality of distinct values for the corresponding type of deterministic characteristic of the iteration.
15. The system of claim 11 , wherein the system is further configured to:
identify at least one failed subpopulation of the plurality of subpopulations for which all potential types of deterministic characteristics have been exhausted, wherein the potential types of deterministic characteristics are a predetermined set of types of characteristics to be used for partitioning the population, wherein all potential types of deterministic characteristics have been exhausted for a failed subpopulation of the plurality of subpopulations when the failed subpopulation was created by splitting the device activity data once using each of the potential types of deterministic characteristics; and
create at least one clustering partition by applying a clustering model to at least one portion of the device activity data corresponding to devices of the at least one failed subpopulation, wherein the plurality of partitions includes the at least one clustering partition.
16. The system of claim 11 , wherein the system is further configured to:
generate a distance matrix for the candidate subpopulation at each iteration, wherein the density value for each iteration is determined based on the distance matrix generated at the iteration.
17. The system of claim 16 , wherein the system is further configured to:
determine a plurality of median distance values for the plurality of rows of the distance matrix generated at each iteration, wherein the density value for each iteration is determined based on the plurality of median distance values determined for the distance matrix for the iteration.
18. The system of claim 17 , wherein the density value determined for the candidate subpopulation at each iteration is determined as a median of the plurality of median distance values.
19. The system of claim 11 , wherein the system is further configured to:
perform at least one mitigation action with respect to the detected anomaly.