Systems and methods for counterfactual explanation in machine learning models
Embodiments described herein provide a two-stage model-agnostic approach for generating counterfactual explanation via counterfactual feature selection and counterfactual feature optimization. Given a query instance, counterfactual feature selection picks a subset of feature columns and values that can potentially change the prediction and then counterfactual feature optimization determines the best feature value for the selected feature as a counterfactual example.
1. A method for generating a counterfactual example for counterfactual explanation in a machine learning model, the method comprising:
receiving a query instance including a plurality of feature columns and corresponding feature values;
generating, by a machine learning model, a prediction result in response to the query instance;
identifying, from the plurality feature columns, a subset of feature columns associated with corresponding alternative feature values that would have potentially changed the prediction result;
determining an optimal policy that maximizes an expected feedback reward earned for modifying the query instance including changing a feature value of at least one feature column of the subset of feature columns;
determining a number of feature columns of the subset of feature columns and a number of associated feature values according to the determined optimal policy;
constructing a counterfactual example using a portion of the determined number of feature columns and the number of associated feature values; and
outputting, by the machine learning model, the counterfactual example as a counterfactual explanation for the prediction result in response to the query instance, wherein the optimal policy is determined by reinforcement learning (RL) in which:
changing the feature value of the at least one feature column is treated as an action in the RL;
the modified query instance is treated as a state in the RL; and
a reward function of the RL for determining the expected feedback reward is a prediction score of a different label other than the prediction result.
2. The method of claim 1 , wherein the optimal policy is determined subject to a sparsity constraint for selecting the action.
3. The method of claim 2 , wherein the sparsity constraint is imposed via a probability distribution parameterized by parameters of the machine learning model.
4. The method of claim 1 , wherein the optimal policy is determined by a gradientless descent procedure that encourages a sparse solution for selecting the action and a gradient descent procedure that encourages an exploration solution for selecting the action.
5. The method of claim 1 , wherein:
the counterfactual example is a first counterfactual example; and
the constructing comprises constructing a diverse set of counterfactual examples including the first counterfactual example and a second counterfactual example using at least one feature column different from the portion of the determined number of feature columns.
6. The method of claim 5 , further comprising:
sorting the set of counterfactual examples based on respective objective function values; and
removing duplicate counterfactual examples from the set of counterfactual examples.
7. The method of claim 1 , wherein the machine learning model is a multivariate time series classifier that receives the query instance of a time series of input sequences.
8. The method of claim 1 , wherein the subset of feature columns are identified from a number of nearest neighboring query instances to the query instance via a nearest neighbor search tree built from a training dataset.
9. A system for generating a counterfactual example for counterfactual explanation in a machine learning model, the system comprising:
a memory that stores the machine learning model;
a data interface that receives a query instance including a plurality of feature columns and corresponding feature values; and
a processor that reads instructions from the memory to perform:
generating, by the machine learning model, a prediction result in response to the query instance;
identifying, from the plurality feature columns, a subset of feature columns associated with corresponding alternative feature values that would have potentially changed the prediction result;
determining an optimal policy that maximizes an expected feedback reward earned for modifying the query instance including changing a feature value of at least one feature column of the subset of feature columns;
determining a number of feature columns of the subset of feature columns and a number of associated feature values according to the determined optimal policy;
constructing a counterfactual example using a portion of the determined number of feature columns and the number of associated feature values; and
outputting, by the machine learning model, the counterfactual example as a counterfactual explanation for the prediction result in response to the query instance, wherein the optimal policy is determined by reinforcement learning (RL) in which:
changing the feature value of the at least one feature column is treated as an action in the RL;
the modified query instance is treated as a state in the RL; and
a reward function of the RL for determining the expected feedback reward is a prediction score of a different label other than the prediction result.
10. The system of claim 9 , wherein the optimal policy is determined subject to a sparsity constraint that is imposed on a stochastic policy for selecting the action.
11. The system of claim 10 , wherein the sparsity constraint is imposed via a probability distribution parameterized by parameters of the machine learning model.
12. The system of claim 9 , wherein the optimal policy is determined by a gradientless descent procedure that encourages a sparse solution for selecting the action and a gradient descent procedure that encourages an exploration solution for selecting the action.
13. The system of claim 9 , wherein the counterfactual example is a first counterfactual example and the processor further reads instructions from the memory to:
construct a diverse set of counterfactual examples including the first counterfactual example and a second counterfactual example using at least one feature column different from the portion of the determined number of feature columns.
14. The system of claim 13 , wherein the processor further reads instructions from the memory to perform:
sorting the set of counterfactual examples based on respective objective function values; and
removing duplicate counterfactual examples from the set of counterfactual examples.
15. The system of claim 9 , wherein the machine learning model is a multivariate time series classifier that receives the query instance of a time series of input sequences.
16. A processor-readable non-transitory storage medium storing processor-readable instructions for generating a counterfactual example for counterfactual explanation in a machine learning model, the instructions being executed by a processor to perform:
receiving a query instance including a plurality of feature columns and corresponding feature values;
generating, by a machine learning model, a prediction result in response to the query instance;
identifying, from the plurality feature columns, a subset of feature columns associated with corresponding alternative feature values that would have potentially changed the prediction result;
determining an optimal policy that maximizes an expected feedback reward earned for modifying the query instance including changing a feature value of at least one feature column of the subset of feature columns;
determining a number of feature columns of the subset of feature columns and a number of associated feature values according to the determined optimal policy;
constructing a counterfactual example using a portion of the determined number of feature columns and the number of associated feature values; and
outputting, by the machine learning model, the counterfactual example as a counterfactual explanation for the prediction result in response to the query instance, wherein the optimal policy is determined by reinforcement learning (RL) in which:
changing the feature value of the at least one feature column is treated as an action in the RL;
the modified query instance is treated as a state in the RL; and
a reward function of the RL for determining the expected feedback reward is a prediction score of a different label other than the prediction result.
17. The method of claim 1 , further comprising selecting to output the counterfactual example as the counterfactual explanation based on its proximity to the query instance.
18. The method of claim 1 , wherein:
one of the determined number of feature columns is a continuous feature column;
the constructing the counterfactual example includes refining the associated feature value of the continuous feature column using a gradientless descent procedure to optimize the proximity of the counterfactual explanation to the query instance; and
the outputting the counterfactual example includes outputting the proximity-optimized counterfactual explanation as the counterfactual explanation for the prediction result in response to the query instance.
19. The processor-readable non-transitory storage medium of claim 16 , wherein the optimal policy is determined subject to a sparsity constraint for selecting the action.
20. The processor-readable non-transitory storage medium of claim 16 , wherein the subset of feature columns are identified from a number of nearest neighboring query instances to the query instance via a nearest neighbor search tree built from a training dataset.