Abnormality detection based on causal graphs representing causal relationships of abnormalities
An example method for abnormality detection based on causal graphs representing causal relationships of abnormalities includes detecting an abnormality in a test data set and generating a counterfactual data set for the test data set. The method further includes determining a quantitative feature dependence between the test data set and the counterfactual data set and determining a causal relationship of the abnormality based on the quantitative feature dependence. The method also includes generating a causal graph that represents the causal relationship of the abnormality. The method may also implement an action to mitigate the abnormality based on the causal graph.
1 . An apparatus comprising:
a processor; and
a computer-readable medium storing instructions which, when executed by the processor, cause the processor to perform a first plurality of operations comprising:
detecting an abnormality in a test data set via an abnormality detection model;
generating one or more counterfactual data sets for the test data set;
determining a quantitative feature dependence between the test data set and the one or more counterfactual data sets;
determining a causal relationship of the abnormality based on the quantitative feature dependence via a causality discovery algorithm; and
generating a causal graph that represents the causal relationship of the abnormality,
wherein the causality discovery algorithm is a pairwise causality discovery algorithm, and the determining of the causal relationship of the abnormality based on the quantitative feature dependence comprises:
generating, via the abnormality detection model and based on the test data set with the abnormality, a predefined number of counterfactual data sets;
evaluating pairwise causality, via the pairwise causality discovery algorithm, to recognize the causality relationship; and
outputting the causality relationship, and
wherein the abnormality detection model is trained via a second plurality of operations, the second plurality of operations comprising:
generating feature weights based on information comprising feature labelling;
training the abnormality detection model based on the feature weights and training data sets;
removing abnormal data from the training data sets; and
training, after the removing the abnormal data from the training data sets, the abnormality detection model to be a counterfactual generation model that is configured to generate the one or more counterfactual data sets,
wherein the determining the quantitative feature dependence between the test data set and the one or more counterfactual data sets comprises;
calculating a difference in numerical value for each feature of abnormal data of the test data set and median values of the one or more counterfactual data sets;
generating feature contributions of the abnormal data of the test data set and the one or more counterfactual data sets via a feature dependence interpretation model;
generating feature contributions for each numerical difference of features of the abnormal data of the test data set;
eliminating one or more feature dependence interpretations having feature contributions for a numerical difference of features of the abnormal data of the test data set that are less than a threshold; and
transforming the feature contributions for a numerical difference of features of the abnormal data of the test data set that are not less than the threshold into a table that quantitatively describes feature dependence.
2 . The apparatus of claim 1 , wherein the first plurality of operations further comprises:
implementing an action to mitigate the abnormality based on the causal graph.
3 . The apparatus of claim 1 , wherein the detecting the abnormality in the test data set comprises:
acquiring an abnormality score for the test data set via the abnormality detection model;
determining the abnormality score is greater than a threshold; and adding, when the abnormality score is greater than the threshold, the test data set to the abnormality detection model, wherein the abnormality detection model is configured to generate the one or more counterfactual data sets.
4 . The apparatus of claim 1 , wherein the information comprising feature labelling is input by a user or default information.
5 . The apparatus of claim 1 , wherein the feature labelling comprises: a label of a feature of a data set; and
a weight for the feature of a data set.
6 . The apparatus of claim 1 , wherein the generating the one or more counterfactual data sets for the test data set is performed via the abnormality detection model that is configured to generate the one or more counterfactual data sets.
7 . The apparatus of claim 1 , wherein the table that quantitatively describes feature dependence comprises one or more rows, wherein each row of the one or more rows represents an inference that a target feature of the features of the abnormal data of the test data set changes by an amount due to a joint effect of one or more pluralities of features of the features of the abnormal data of the test data set.
8 . The apparatus of claim 1 , wherein the determining the causal relationship of the abnormality based on the quantitative feature dependence further comprises:
obtaining two features of the abnormality in the test data set; and
determining that a causal relationship for the two features does not exist in information comprising feature dependence.
9 . The apparatus of claim 8 , wherein the causality relationship is recognized after iteration of possible feature pairs of a subset of elements in a table that quantitatively describes feature dependence.
10 . The apparatus of claim 8 , wherein the pairwise causality discovery algorithm comprises an additive-noise model (ANM) or an information geometric causal inference (IGCI) model.
11 . The apparatus of claim 1 , wherein the determining the causal relationship of the abnormality based on the quantitative feature dependence comprises:
obtaining a plurality of features of the abnormality in the test data set;
determining that a causal relationship for the plurality of features exists in information comprising feature dependence; and
outputting the causal relationship.
12 . The apparatus of claim 1 , wherein the causal relationship represents the quantitative feature dependence of the abnormality.
13 . The apparatus of claim 1 , wherein the feature dependence interpretation model is trained via a third plurality of operations, the third plurality of operations comprising:
determining, from information comprising feature labelling, a portion of the information to be used to reinforce generalization performance of the apparatus;
acquiring one or more minimum correlated feature subsets;
training the feature dependence interpretation model based on the one or more minimum correlated feature subsets; and
training the feature dependence interpretation model based on a randomly selected subset of the training data set.
14 . The apparatus of claim 13 , wherein the one or more minimum correlated feature subsets are determined via a fourth plurality of operations, the fourth plurality of operations comprising:
determining a correlation metric that is able to evaluate nonlinear correlations;
applying the correlation metric to pairwise features of second training data sets to calculate correlation scores for the pairwise features;
accumulating a numeric value of pairwise features that have a correlation score greater than a correlation threshold based on the correlation scores for the pairwise features; and
extracting the pairwise features having a correlation score greater than the correlation threshold into the one or more minimum correlated feature subsets.
15 . The apparatus of claim 14 , wherein the extracting the pairwise features having a correlation score greater than the correlation threshold into the one or more minimum correlated feature subsets comprises:
iteratively extracting, into the one or more minimum correlated feature subsets, a feature of the pairwise features having a correlation score greater than the correlation threshold that has the greatest number of correlated features, until the numeric value outside the one or more minimum correlated feature subset is not greater than two.
16 . The apparatus of claim 14 , wherein the fourth plurality of operations further comprises:
outputting at least one of:
the one or more minimum correlated feature subsets; and a correlation matrix.
17 . A method comprising:
detecting, by a processor, an abnormality in a test data set via an abnormality detection model;
generating, by the processor, one or more counterfactual data sets for the test data set;
determining, by the processor, a quantitative feature dependence between the test data set and the one or more counterfactual data sets;
determining, by the processor, a causal relationship of the abnormality based on the quantitative feature dependence via a causality discovery algorithm;
generating, by the processor, a causal graph that represents the causal relationship of the abnormality,
wherein the causality discovery algorithm is a pairwise causality discovery algorithm, and the determining of the causal relationship of the abnormality based on the quantitative feature dependence comprises:
generating, via the abnormality detection model and based on the test data set with the abnormality, a predefined number of counterfactual data sets;
evaluating pairwise causality, via the pairwise causality discovery algorithm, to recognize the causality relationship; and
outputting the causality relationship, and
wherein the abnormality detection model is trained via a second plurality of operations, the second plurality of operations comprising:
generating feature weights based on information comprising feature labelling;
training the abnormality detection model based on the feature weights and training data sets;
removing abnormal data from the training data sets; and
training, after removing the abnormal data from the training data sets, the abnormality detection model to be a counterfactual generation model that is configured to generate the one or more counterfactual data sets,
wherein the determining the quantitative feature dependence between the test data set and the one or more counterfactual data sets comprises:
calculating a difference in numerical value for each feature of abnormal data of the test data set and median values of the one or more counterfactual data sets;
generating feature contributions of the abnormal data of the test data set and the one or more counterfactual data sets via a feature dependence interpretation model;
generating feature contributions for each numerical difference of features of the abnormal data of the test data set;
eliminating one or more feature dependence interpretations having feature contributions for a numerical difference of features of the abnormal data of the test data set that are less than a threshold; and
transforming the feature contributions for a numerical difference of features of the abnormal data of the test data set that are not less than the threshold into a table that quantitatively describes feature dependence.
18 . A non-transitory computer-readable storage device storing a plurality of instructions which, when executed by a processor, cause the processor to perform operations, the operations comprising:
detecting an abnormality in a test data set via an abnormality detection model;
generating one or more counterfactual data sets for the test data set;
determining a quantitative feature dependence between the test data set and the one or more counterfactual data sets;
determining a causal relationship of the abnormality based on the quantitative feature dependence via a causality discovery algorithm; and
generating a causal graph that represents the causal relationship of the abnormality,
wherein the causality discovery algorithm is a pairwise causality discovery algorithm, and the determining of the causal relationship of the abnormality based on the quantitative feature dependence comprises:
generating, via the abnormality detection model and based on the test data set with the abnormality, a predefined number of counterfactual data sets;
evaluating pairwise causality, via the pairwise causality discovery algorithm, to recognize the causality relationship; and
outputting the causality relationship, and
wherein the abnormality detection model is trained via a second plurality of operations, the second plurality of operations comprising:
generating feature weights based on information comprising feature labelling;
training the abnormality detection model based on the feature weights and training data sets;
removing abnormal data from the training data sets; and
training, after removing the abnormal data from the training data sets, the abnormality detection model to be a counterfactual generation model that is configured to generate the one or more counterfactual data sets,
wherein the determining the quantitative feature dependence between the test data set and the one or more counterfactual data sets comprises;
calculating a difference in numerical value for each feature of abnormal data of the test data set and median values of the one or more counterfactual data sets;
generating feature contributions of the abnormal data of the test data set and the one or more counterfactual data sets via a feature dependence interpretation model;
generating feature contributions for each numerical difference of features of the abnormal data of the test data set,
eliminating one or more feature dependence interpretations having feature contributions for a numerical difference of features of the abnormal data of the test data set that are less than a threshold; and
transforming the feature contributions for a numerical difference of features of the abnormal data of the test data set that are not less than the threshold into a table that quantitatively describes feature dependence.