Anomaly detection in streaming telephone network data
In one example, a method includes receiving a feature vector that characterizes a call history for a telephone network subscriber, wherein the feature vector comprises respective categorical values for one or more categorical features and respective continuous values for one or more continuous features, and applying, to the categorical values, a first algorithm to determine a categorical score for the feature vector. The example method further includes applying, to the continuous values, an isolation forest algorithm to determine a continuous score for the feature vector, and outputting, in response to determining at least one of the categorical score for the feature vector and the continuous score for the feature vector indicate the feature vector is anomalous, an indication that the feature vector is anomalous.
1. A computing device comprising:
one or more programmable processors operably coupled to a memory, the memory configured to cause the one or more programmable processors to:
receive a feature vector that characterizes a call history for a telephone network subscriber, wherein the feature vector comprises respective categorical values for one or more categorical features and respective continuous values for one or more continuous features;
apply, to the categorical values, a first algorithm to determine a categorical score for the feature vector;
apply, to the continuous values, an isolation forest algorithm to determine a continuous score for the feature vector; and
output, in response to determining at least one of the categorical score for the feature vector and the continuous score for the feature vector indicates the feature vector is anomalous, an indication that the feature vector is anomalous.
2. The computing device of claim 1 ,
wherein a categorical itemset d from possible categorical itemsets D for the one or more categorical features comprises a combination of respective categorical values for at least one of the one or more categorical features,
wherein to apply the isolation forest algorithm, the memory is configured to cause the one or more programmable processors to:
apply, to the continuous values, respective isolation forest models fit for categorical itemsets D g from D to determine respective anomaly scores for the categorical itemsets D g ; and
combine the respective anomaly scores for the categorical itemsets D g to determine the continuous score for the feature vector.
3. The computing device of claim 2 , wherein to apply the isolation forest algorithm, the memory is configured to cause the one or more programmable processors to:
determine, from possible categorical itemsets D, the categorical itemsets D g that include each categorical itemset d having all of one or more mandatory categorical features.
4. The computing device of claim 3 , wherein the one or more mandatory categorical features are user-specified.
5. The computing device of claim 3 , wherein to apply the isolation forest algorithm, the memory is configured to cause the one or more programmable processors to:
determine, from a plurality of possible categorical itemsets D, the categorical itemsets D g as only including each categorical itemset d that:
(1) includes all of the one or more mandatory categorical features; and
(2) has a number of categorical values that satisfies a maximum length parameter.
6. The computing device of claim 2 , wherein the memory is configured to cause the one or more programmable processors to:
determine, from a training set, one or more training feature vectors that each has a categorical itemset that has a number of matching training feature vectors from the training set to satisfy a low support parameter; and
fit each isolation forest model of the isolation forest models to respective continuous values for one or more continuous features of the determined one or more training feature vectors that each includes all of the one or more categorical values of the corresponding categorical itemset from the categorical itemsets D g for the isolation forest model.
7. The computing device of claim 2 , wherein the memory is configured to cause the one or more programmable processors to:
fit each isolation forest model of the isolation forest models to respective continuous values for one or more continuous features of one or more training feature vectors that each includes all of the one or more categorical values of the corresponding categorical itemset from the categorical itemsets D g for the isolation forest model.
8. The computing device of claim 2 , wherein the memory is configured to cause the one or more programmable processors to:
fit each isolation forest model of the isolation forest models to one or more training feature vectors according to:
X d ={x i ϵX|x i c ⊃ d}, where X d is the set of training feature vectors containing a categorical itemset d; and
iforest d =train_model(X d q ), where X d q are the continuation values for one or more continuous features of X d .
9. The computing device of claim 2 , wherein the memory is configured to cause the one or more programmable processors to apply the isolation forest algorithm according to:
score
cont
(
x
i
)
=
1
d
∈
D
g
⩓
x
i
c
⊇
d
∑
d
∈
D
g
⩓
x
i
c
⊇
d
[
iforest
d
(
x
i
q
)
]
,
where x i c represents the categorical values of the instance to be scored,
where x i represents the feature vector, and
where x i q represents the continuous values of the feature vector.
10. The computing device of claim 2 , wherein the memory is configured to cause the one or more programmable processors to:
determine D g according to
D g ={d:dϵD|s d ⊃ MAN_FEATS | d |≤MAX_LEN},
where s d is the subspace to which categorical itemset d belongs,
where MAN_FEATS specifies one or more mandatory categorical features,
where |d| is a number of categories values in the categorical itemset d, and
where MAX_LEN specifies a number of category values.
11. The computing device of claim 2 , wherein the memory is configured to cause the one or more programmable processors to:
fit each isolation forest model of the isolation forest models to respective continuous values for one or more continuous features of one or more training feature vectors that each includes all of the one or more categorical values of the corresponding categorical itemset from the categorical itemsets D g for the isolation forest model;
determine, for each isolation forest model of the isolation forest models, a corresponding mean vector for the respective continuous values for the one or more continuous features of the one or more training feature vectors that each includes all of the one or more categorical values of the corresponding categorical itemset from the categorical itemsets D g for the isolation forest model;
determine a baseline anomaly score by applying the isolation forest algorithm to the mean vector;
for each continuous value of the continuous values of the feature vector, substituting the continuous value to a corresponding position in the mean vector to obtain a modified mean vector and determining a per-feature score for the continuous value by applying the isolation forest algorithm to the modified mean vector; and
output each per-feature score for each continuous value.
12. The computing device of claim 1 , wherein the memory is configured to cause the one or more programmable processors to apply, to the categorical values, the first algorithm to determine the categorical score for the feature vector according to:
score
cat
(
x
i
)
=
∑
d
⊆
x
i
c
⩓
d
∈
D
p
(
1
supp
(
d
)
d
)
,
where x i c represents the categorical values of the feature vector,
where x i represents the feature vector,
wherein a categorical itemset d from all possible categorical itemsets D for the one or more categorical features comprises a combination of respective categorical values for at least one of the one or more categorical features,
where D p is a set of infrequent itemsets within a training dataset X,
where |d| is a number of categories values in the categorical itemset d,
where supp(d) is a support of itemset d within a training dataset X.
13. The computing device of claim 12 , wherein each categorical itemset d i from D p satisifies:
(1) supp(d i )<SIGMA, where SIGMA is a user-specified threshold;
(2) |d i |<MAX_LEN, wherein MAX_LEN is a user-specified threshold; and
(3) d j d i |d j ϵD p , d j ≠d i .
14. The computing device of claim 1 , wherein the continuous features comprise at least one of a call duration group, a call density group, a call direction, a call location group, and a customer details group.
15. The computing device of claim 1 , wherein the categorical features comprise at least one of a provider affiliation, a customer type, a customer segment, and a customer status.
16. The computing device of claim 1 ,
wherein a categorical itemset d from all possible categorical itemsets D for the one or more categorical features comprises a combination of respective categorical values for at least one of the one or more categorical features,
wherein to apply the first algorithm, the memory is configured to cause the one or more programmable processors to:
determine infrequent categorical itemsets D p within a training dataset X;
for each infrequent categorical itemset from the infrequent categorical itemsets D p , determine a corresponding number of training feature vectors in the training dataset X having the infrequent categorical itemset;
determine, for each infrequent categorical itemset d i from infrequent categorical itemsets D p having categorical values that are a subset of the categorical values of the feature vector, respective anomaly scores based at least on the corresponding number of training feature vectors and a corresponding length for the infrequent categorical itemset d i .
17. The computing device of claim 16 , wherein the memory is configured to cause the one or more programmable processors to:
for a categorical feature of the categorical features, accumulate anomaly scores determined for each infrequent categorical itemset d i that includes the categorical feature to obtain a feature importance of the categorical feature to the feature vector; and
output an indication of the feature importance.
18. The computing device of claim 1 , wherein the first algorithm comprises an outlier detection algorithm for mixed-attribute data (ODMAD).
19. The computing device of claim 1 , wherein the computing device comprises one of a cloud-based computing system, one or more servers, one or more workstations, a computing system executing virtual machines or containers, a mobile computing device, or a mobile phone.
20. The computing device of claim 1 , wherein the memory is configured to cause the one or more programmable processors to:
apply, in response to determining at least one of the categorical score for the feature vector or the continuous score for the feature vector indicate the feature vector is anomalous, a filter based on z-scores for features of network usage data for the telephone network subscriber; and
output the indication that the feature vector is anomalous in response to determining the filter indicates the feature vector is a true positive.
21. A method comprising:
receiving a feature vector that characterizes a call history for a telephone network subscriber, wherein the feature vector comprises respective categorical values for one or more categorical features and respective continuous values for one or more continuous features;
applying, to the categorical values, a first algorithm to determine a categorical score for the feature vector;
applying, to the continuous values, an isolation forest algorithm to determine a continuous score for the feature vector; and
outputting, in response to determining at least one of the categorical score for the feature vector and the continuous score for the feature vector indicates the feature vector is anomalous, an indication that the feature vector is anomalous.
22. The method of claim 21 ,
wherein a categorical itemset d from possible categorical itemsets D for the one or more categorical features comprises a combination of respective categorical values for at least one of the one or more categorical features,
wherein applying the isolation forest algorithm comprises:
applying, to the continuous values, respective isolation forest models fit for categorical itemsets D g from D to determine respective anomaly scores for the categorical itemsets D g ; and
combining the respective anomaly scores for the categorical itemsets D g to determine the continuous score for the feature vector.
23. The method of claim 22 , wherein applying the isolation forest algorithm comprises determining, from possible categorical itemsets D, the categorical itemsets D g that include each categorical itemset d having all of one or more mandatory categorical features.
24. The method of claim 23 , wherein the one or more mandatory categorical features are user-specified.
25. A non-transitory computer-readable medium comprising instructions for causing at least one programmable processor to:
receive a feature vector that characterizes a call history for a telephone network subscriber, wherein the feature vector comprises respective categorical values for one or more categorical features and respective continuous values for one or more continuous features;
apply, to the categorical values, a first algorithm to determine a categorical score for the feature vector;
apply, to the continuous values, an isolation forest algorithm to determine a continuous score for the feature vector; and
output, in response to determining at least one of the categorical score for the feature vector and the continuous score for the feature vector indicates the feature vector is anomalous, an indication that the feature vector is anomalous.