IP Library › Granted Patent US 12,321,826
Granted Patent B2
US 12,321,826 · App. 17/322,828 · Granted Jun 3, 2025

System and method for utilizing grouped partial dependence plots and game-theoretic concepts and their extensions in the generation of adverse action reason codes

Inventors: Alexey Miroshnikov (Evanston, IL); Konstandinos Kotsiopoulos (Easthampton, MA); Arjun Ravi Kannan (Buffalo Grove, IL); Raghu Kulkarni (Buffalo Grove, IL); Steven Dickerson (Deerfield, IL)
Assignee: Discover Financial Services
G06N20/00G06F17/16G06F18/24137G06N3/084G06N5/045G06V10/7753
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,321,826
App. No.
17/322,828
Granted
Jun 3, 2025
Kind
B2
Abstract

A framework for interpreting machine learning models is proposed that utilizes interpretability methods to determine the contribution of groups of input variables to the output of the model. Input variables are grouped based on dependencies with other input variables. The groups are identified by processing a training data set with a clustering algorithm. Once the groups of input variables are defined, scores related to each group of input variables for a given instance of the input vector processed by the model are calculated according to one or more algorithms. The algorithms can utilize group Partial Dependence Plot (PDP) values, Shapley Additive Explanations (SHAP) values, and Banzhaf values, and their extensions among others, and a score for each group can be calculated for a given instance of an input vector per group. These scores can then be sorted, ranked, and then combined into one hybrid ranking.

Claims (81)

1. A method, comprising:

training a machine learning (ML) model by carrying out a machine learning process on a training dataset comprising a set of input vectors and a corresponding set of output values, wherein the trained machine learning model is configured to (i) receive an input vector comprising respective values for a given set of input variables and (ii) based on an evaluation of the received input vector, generate a prediction;

defining a plurality of variable groups based on an evaluation of dependencies between input variables in the given set of input variables of the ML model, wherein (i) each variable group includes at least one input variable from the given set of input variables and (ii) each of at least a subset of the variable groups includes two or more input variables from the given set of input variables that are determined to have a threshold level of dependence with one another;

after training the machine learning model and dividing the given set of input variables into the plurality of variable groups:

receiving a given input vector comprising respective values for the given set of input variables;

processing, by the ML model, the given input vector to generate an output vector a given prediction corresponding to the given input vector;

evaluating the given input vector and the given prediction using (i) the defined plurality of variable groups and (ii) at least one model interpretability technique that serves to explain the ML model by quantifying contributions of the given set of input variables to predictions generated by the ML model;

based on the evaluating, generating a respective score for each variable group in the defined plurality of variable groups, wherein the respective score for each variable group quantifies a respective contribution of the variable group to the given prediction; and

based on the respective scores determined for the defined plurality of variable groups, identifying one or more input variables of the ML model that contributed most to the given prediction.

2. The method of claim 1 , wherein defining the plurality of variable groups comprises dividing the given set of input variables into the plurality of variable groups based on a clustering algorithm applied to a training data set comprising a number of instances of the input vector and corresponding target output values.

3. The method of claim 1 , wherein the respective score for each variable group is calculated based on at least one of:

group partial dependence plot (PDP) values;

Shapley Additive Explanation (SHAP) values;

Banzhaf values;

quotient game SHAP values;

two-step SHAP value;

Owen values;

Banzhaf-Owen values; or

symmetric coalitional Banzhaf values.

4. The method of claim 3 , wherein the respective score for each variable group comprises a respective hybrid score generated based on at least two of the group PDP values, the SHAP values, the Banzhaf values, the quotient game SHAP values, the two-step SHAP values, the Owen values, the Banzhaf-Owen values, or the symmetric coalitional Banzhaf values.

5. The method of claim 1 , further comprising:

generating one or more adverse action reason codes corresponding to the identified one or more input variables.

6. The method of claim 5 , wherein the given prediction represents a determination related to a consumer's credit, the method further comprising generating, at a server device associated with a financial service provider, a communication to transmit to a device associated with the consumer, wherein the communication includes information corresponding to the one or more adverse action reason codes.

7. The method of claim 1 , wherein the respective score for each of at least the subset of variable groups is generated by performing a multivariate interpolation of a number of sample points in a corresponding partial dependence plot (PDP) table.

8. The method of claim 1 , wherein the ML model comprises one of:

a neural network model; a linear or logistic regression model; or a gradient boosting machine model.

9. The method of claim 1 , wherein the respective score for each variable group comprises a respective hybrid score generated by aggregating:

a respective first score that is based on one or more partial dependence plot (PDP) tables; and

a respective second score that is based on a Shapley Additive Explanation (SHAP) value or Banzhaf value.

10. A computing system comprising:

a memory;

one or more processors; and

program instructions stored in the memory that, when executed by the one or more processors, cause the computing system to:

train a machine learning (ML) model by carrying out a machine learning process on a training dataset comprising a set of input vectors and a corresponding set of output values, wherein the trained machine learning model is configured to (i) receive an input vector comprising respective values for a given set of input variables and (ii) based on an evaluation of the received input vector, generate a prediction;

define a plurality of variable groups based on an evaluation of dependencies between input variables in the given set of input variables of the ML model, wherein (i) each variable group includes at least one input variable from the given set of input variables and (ii) each of at least a subset of the variable groups includes two or more input variables from the given set of input variables that are determined to have a threshold level of dependence with one another;

after training the machine learning model and dividing the given set of input variables into the plurality of variable groups:

receive a given input vector comprising respective values for the given set of input variables;

process, by the ML model, the given input vector to generate a given prediction corresponding to the given input vector;

evaluate the given input vector and the given prediction using (i) the defined plurality of variable groups and (ii) at least one model interpretability technique that serves to explain the ML model by quantifying contributions of the given set of input variables to predictions generated by the ML model;

based on the evaluating, generate a respective score for each variable group in the defined plurality of variable groups, wherein the respective score for each variable group quantifies a respective contribution of the variable group to the given prediction; and

based on the respective scores determined for the defined plurality of variable groups, identify one or more input variables of the ML model that contributed most to the given prediction.

11. The computing system of claim 10 , wherein the plurality of variable groups are defined by dividing the given set of input variables into the plurality of variable groups based on a clustering algorithm applied to a training data set comprising a number of instances of the input vector and corresponding target output values.

12. The computing system of claim 10 , wherein the respective score for each group of variable group is calculated based on at least one of:

group partial dependence plot (PDP) values;

Shapley Additive Explanation (SHAP) values;

Banzhaf values;

quotient game SHAP values;

two-step SHAP value;

Owen values;

Banzhaf-Owen values; or

symmetric coalitional Banzhaf values.

13. The computing system of claim 12 , wherein the respective score for each variable group comprises a respective hybrid score generated based on at least two of the group PDP values, the SHAP values, the Banzhaf values, the quotient game SHAP values, the two-step SHAP values, the Owen values, the Banzhaf-Owen values, or the symmetric coalitional Banzhaf values.

14. The computing system of claim 10 , further comprising program instructions stored in the memory that, when executed by the one or more processors, cause the computing system to:

generate one or more adverse action reason codes corresponding to the identified one or more input variables.

15. The computing system of claim 14 , wherein the given prediction represents a determination related to a consumer's credit, and wherein the computing system further comprises program instructions stored in the memory that, when executed by the one or more processors, cause the computing system to:

generate, at a server device associated with a financial service provider, a communication to transmit to a device associated with the consumer, wherein the communication includes information corresponding to the one or more adverse action reason codes.

16. The computing system of claim 10 , wherein the respective score for each of at least the subset of variable groups is generated by performing a multivariate interpolation of a number of sample points in a corresponding partial dependence plot (PDP) table.

17. The computing system of claim 10 , wherein at least one processor of the one or more processors and the memory are included in a server device configured to implement a service, and wherein the service is configured to receive a request to process a credit application and, responsive to determining that the credit application is denied, generate one or more adverse action reason codes associated with the credit application.

18. The computing system of claim 10 , wherein the respective score for each variable group comprises a respective hybrid score generated by aggregating:

a respective first score that is based on a one or more partial dependence plot (PDP) tables; and

a respective second score that is based on a Shapley Additive Explanation (SHAP) value or Banzhaf value.

19. A non-transitory computer-readable medium storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:

training a machine learning (ML) model by carrying out a machine learning process on a training dataset comprising a set of input vectors and a corresponding set of output values, wherein the trained machine learning model is configured to (i) receive an input vector comprising respective values for a given set of input variables and (ii) based on an evaluation of the received input vector, generate a prediction;

defining a plurality of variable groups based on an evaluation of dependencies between input variables in the given set of input variables of the ML model, wherein (i) each variable group includes at least one input variable from the given set of input variables and (ii) each of at least a subset of the variable groups includes two or more input variables from the given set of input variables that are determined to have a threshold level of dependence with one another;

after training the machine learning model and dividing the given set of input variables into the plurality of variable groups:

receiving a given input vector comprising respective values for the given set of input variables;

processing, by the ML model, the given input vector to generate a given prediction corresponding to the given input vector;

evaluating the given input vector and the given prediction using (i) the defined plurality of variable groups and (ii) at least one model interpretability technique that serves to explain the ML model by quantifying contributions of the given set of input variables to predictions generated by the ML model;

based on the evaluating, generating a respective score for each variable group in the defined plurality of variable groups, wherein the respective score for each variable group quantifies a respective contribution of the variable group to the given prediction; and

based on the respective scores determined for the defined plurality of variable groups, identifying one or more input variables of the ML model that contributed most to the given prediction.

20. The non-transitory computer readable medium of claim 19 , wherein defining the plurality of variable groups comprises dividing the given set of input variables into the plurality of variable groups based on a clustering algorithm applied to a training data set comprising a number of instances of the input vector and corresponding target output values.

21. The non-transitory computer readable medium of claim 19 , wherein the respective score for each variable group is calculated based on at least one of:

group partial dependence plot (PDP) values;

Shapley Additive Explanation (SHAP) values;

Banzhaf values;

quotient game SHAP values;

two-step SHAP value;

Owen values;

Banzhaf-Owen values; or

symmetric coalitional Banzhaf values.

22. The non-transitory computer readable medium of claim 21 , wherein the respective score for each variable group comprises a respective hybrid score generated based on at least two of the group PDP values, the SHAP values, the Banzhaf values, the quotient game SHAP values, the two-step SHAP values, the Owen values, the Banzhaf-Owen values, or the symmetric coalitional Banzhaf values.

Assignments (2)
MERGER Recorded Jul 2, 2025
From: DISCOVER FINANCIAL SERVICES
To: CAPITAL ONE FINANCIAL CORPORATION
Reel/Frame 071784/0903 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 17, 2021
From: MIROSHNIKOV, ALEXEY; KOTSIOPOULOS, KONSTANDINOS; KANNAN, ARJUN RAVI; KULKARNI, RAGHU; DICKERSON, STEVEN
To: DISCOVER FINANCIAL SERVICES
Reel/Frame 056265/0900 →
Continuity (2)
Continuation In Part 16868019 · May 6, 2020
Related Publication 20210383275A1 · Dec 9, 2021
References Cited (174)
US 8370253B1 · Grossman · 2013 [cited by examiner]
US 11030583B1 · Garg et al. · 2021 [cited by applicant]
US 20110270782A1 · Trenner et al. · 2011 [cited by applicant]
US 20140229415A1 · Martineau et al. · 2014 [cited by applicant]
US 20180285685A1 · Singh et al. · 2018 [cited by applicant]
US 20190087744A1 · Schiemenz · 2019 [cited by applicant]
US 20190188605A1 · Zavesky et al. · 2019 [cited by applicant]
US 20190279111A1 · Merrill et al. · 2019 [cited by applicant]
US 20190378210A1 · Merrill et al. · 2019 [cited by applicant]
US 20200081865A1 · Farrar et al. · 2020 [cited by applicant]
US 20200082299A1 · Affonso et al. · 2020 [cited by applicant]
US 20200184350A1 · Bhide et al. · 2020 [cited by applicant]
US 20200218987A1 · Jiang et al. · 2020 [cited by applicant]
US 20200302309A1 · Golding · 2020 [cited by applicant]
US 20200302524A1 · Kamkar · 2020 [cited by examiner]
US 20200320428A1 · Chaloulos et al. · 2020 [cited by applicant]
US 20200372035A1 · Tristan et al. · 2020 [cited by applicant]
US 20200372304A1 · Kenthapadi et al. · 2020 [cited by applicant]
US 20200372406A1 · Wick · 2020 [cited by applicant]
US 20200380398A1 · Weirder et al. · 2020 [cited by applicant]
US 20210133870A1 · Kamkar et al. · 2021 [cited by applicant]
US 20210174222A1 · Dodwell et al. · 2021 [cited by applicant]
US 20210224605A1 · Zhang et al. · 2021 [cited by applicant]
US 20210241033A1 · Yang · 2021 [cited by applicant]
US 20210248503A1 · Hickety et al. · 2021 [cited by applicant]
US 20210256832A1 · Weisz et al. · 2021 [cited by applicant]
US 20210304039A1 · Tang · 2021 [cited by examiner]
US 20210334654A1 · Himanshi et al. · 2021 [cited by applicant]
US 20210350272A1 · Miroshnikov et al. · 2021 [cited by applicant]
US 20210383268A1 · Miroshnikov et al. · 2021 [cited by applicant]
US 20220036203A1 · Nachum et al. · 2022 [cited by applicant]
WO 2021074491A1 · 2021 [cited by applicant]
Kwak, Nojun, and Chong-Ho Choi. “Input feature selection for classification problems.” IEEE transactions on neural networks 13.1 (2002): 143-159. (Year: 2002). [cited by examiner]
Ding, Ruiqiang, and Jianping Li. “Comparisons of two ensemble mean methods in measuring the average error growth and the predictability.” Acta Meteorologica Sinica 25.4 (2011): 395-404. (Year: 2011). [cited by examiner]
Krivoruchko, Esri, and Nathan Wood. “Using Multivariate Interpolation for Estimating Well Performance.” Esri.com (2014) (Year: 2014). [cited by examiner]
Hu, Linwei, et al. “Locally interpretable models and effects based on supervised partitioning (LIME-SUP).” arXiv preprint arXiv: 1806.00663 (2018). (Year: 2018). [cited by examiner]
Lundberg, Scott M., Gabriel G. Erion, and Su-In Lee. “Consistent individualized feature attribution for tree ensembles.” arXiv preprint arXiv: 1802.03888 (2018). (Year: 2018). [cited by examiner]
Manu Joseph. “Interpretability Cracking open the black box—Part III.” Deep & Shallow (2019). (Year: 2019). [cited by examiner]
Chen, Chia-Hsiu, et al. “Comparison and improvement of the predictability and interpretability with ensemble learning models in QSPR applications.” Journal of cheminformatics 12 (2020): 1-16. (Year: 2020). [cited by examiner]
Casalicchio, Giuseppe, Christoph Molnar, and Bernd Bischl. “Visualizing the feature importance for black box models.” Machine Learning and Knowledge Discovery in Databases: Proceedings, Part I 18. Springer International… [cited by examiner]
International Search Report and Written Opinion issued in International Application No. PCT/US2022/029670, mailed on Sep. 7, 2022, 10 pages. [cited by applicant]
Jiang et al. Wasserstein Fair Classification. Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, arXiv: 1907.12059v1 [stat.ML] Jul. 28, 2019, 15 pages. <URL: chrome-extension://efaidnb… [cited by applicant]
Wei et al. Optimized Score Transformation for Consistent Fair Classication, arXiv: 1906.00066v3 [cs.LG] Oct. 29, 2021, 78 pages <URL: chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://arxiv.org/pdf/1906.00066.… [cited by applicant]
Iosifidis et al. FAE: A Fairness-Aware Ensemble Framework, IEEE, arXiv:2002.00695v1 [cs.AI] Feb. 3, 2020, 6 pages. <URL:https://arxiv.org/abs/2002.00695>. [cited by applicant]
Balashankar et al. Pareto-Efficient Fairness for Skewed Subgroup Data. In the International Conference on Machine Learning AI for Social Good Workshop. vol. 8. Long Beach, United States, 2019, 8 pages. [cited by applicant]
Barocas et al. Fairness and Machine Learning: Limitations and Opportunities. [online], 253 pages, 1999 [retrieved online on May 26, 2022]. Retrieved from the Internet:<URL:https://fairmlbook.org/>. [cited by applicant]
Bergstra et al. Algorithms for Hyper-Parameter Optimization. NIPS 11: Proceedings of the 24th International Conference on Neural Information Processing Systems Dec. 2011, 9 pages. [cited by applicant]
Bousquet et al. Stability and Generalization. Journal of Machine Learning Research 2. Mar. 2002, pp. 499-526. [cited by applicant]
Del Barrio et al. Obtaining fairness using optimal transport theory. arXiv preprint arXiv: 1806.03195v2. Jul. 19, 2018, 25 pages. [cited by applicant]
Del Barrio et al. On Approximate Validation of Models: A Kolmogorov-Smirnov Based Approach. arXiv: 1903.08687v1, Mar. 20, 2019, 32 pages. [cited by applicant]
Dwork et al. Generalization in Adaptive Data Analysis and Holdout Reuse. NIPS'15: Proceedings of the 28th International Conference on Neural Information Processing Systems, vol. 2, Dec. 2015, 9 pages. [cited by applicant]
Dwork et al., “Fairness Through Awareness” Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, 214-226 (2012). [cited by applicant]
Equal Employment Opportunity Act. 2 pages [online]. Retrieved online: URL<https://www.dol.gov/sites/dolgov/files/pfccp/regs/compliance/posters/pdf/eeopost.pdf>. [cited by applicant]
Fair Housing Act (FHA). FDIC Law, Regulations, Related Actions, 3 pages [online]. Retrieved online: URL< https://www.ecfr.gov/current/title-12/chapter-III/subchapter-B/part-338. [cited by applicant]
Feldman et al., “Certifying and Removing Disparate Impact” Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 259-268 (2015). [cited by applicant]
Hall et al. A United States Fair Lending Perspective on Machine Learning. Frontiers in Artificial Intelligence. doi: 10.3389/frai.2021.695301, Jun. 7, 2021, 9 pages. [cited by applicant]
Hardt et al., “Equality of Opportunity in Supervised Learning” Advances in Neural Information Processing Systems, 3315-3323 (2015). [cited by applicant]
Hashimoto et al. Fairness Without Demographics in Repeated Loss Minimization. In ICML, 2018, 10 pages. [cited by applicant]
Jiang et al. Identifying and Correcting Label Bias in Machine Learning. Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS). 2020, 10 pages. [cited by applicant]
Jiang et al. Smooth Isotonic Regression: A New Method to Calibrate Predictive Models. AMIA Joint Summits on Translational Science proceedings. 2011, pp. 16-20. [cited by applicant]
Kamiran et al. Classifying without Discriminating. 2009 2nd International Conference on Computer, Control and Communication. doi: 10.1109/IC4.2009.4909197. 2009, pp. 1-6. [cited by applicant]
Kamiran et al. Data Preprocessing Techniques for Classification Without Discrimination. Knowl. Inf. Syst. DOI 10.1007/s10115-011-0463-8. Dec. 3, 2011, 33 pages. [cited by applicant]
Kamiran et al. Discrimination Aware Decision Tree Learning. 2010 IEEE International Conference on Data Mining. doi: 10.1109/ICDM.2010.50. 2010, pp. 869-874. [cited by applicant]
Kamishima et al. Fairness-Aware Classifier with Prejudice Remover Regularizer. Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECMLPKDD), Part … [cited by applicant]
Karush, William. Minima of Functions of Several Variables with Inequalities as Side Constraints. A Dissertation Submitted to the Faculty of the Division of the Physical Sciences in Candidacy for the Degree of Master in … [cited by applicant]
Kearns et al. Algorithmic Stability and Sanity-Check Bounds for Leave-One-Out Cross-Validation. Neural Computation, vol. 11, No. 6. 1999, pp. 1427-1453. [cited by applicant]
Koralov et al. Theory of Probability and Random Processes. Second Edition. Springer. 1998, 349 pages. [cited by applicant]
Kovalev et al. A robust algorithm for explaining unreliable machine learning survival models using the Kolmogorov-Smirnov bounds. Neural Networks. arXiv:2005.02249v1, May 5, 2020. pp. 1-39. [cited by applicant]
Kuhn et al. Nonlinear programming. In Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability. Berkeley, Calif. University of California Press. 90. 1951, pp. 481-492. [cited by applicant]
Lahoti et al. Fairness without Demographics through Adversarially Reweighted Learning. arXiv preprint arXiv:2006.13114. 2020, 13 pages. [cited by applicant]
Markowitz, Harry. Portfolio Selection. The Journal of Finance. vol. 7, No. 1. 1952, pp. 77-91. [cited by applicant]
Pareto. Optimality. In: Multiobjective Linear Programming. Springer International Publishing. DOI 10.1007/978-3-319-21091-9_4. 2016, pp. 85-118. [cited by applicant]
Perrone et al. Fair Bayesian Optimization. AIES '21, May 19-21, 2021, Virtual Event, USA. pp. 854-863. [cited by applicant]
Rawls, John. Justice as Fairness: A Restatement. Harvard University Press. 2001, 32 pages. [cited by applicant]
Royden et al. Real Analysis. Boston: Prentice Hall, 4th Edition. 2010, 516 pages. [cited by applicant]
Santambrogio, Filippo. Optimal Transport for Applied Mathematicians. Calculus of Variations, PDEs and Modeling. Springer. May 2015, 356 pages. [cited by applicant]
Schmidt et al. An Introduction to Artificial Intelligence and Solutions to the Problems of Algorithmic Discrimination. arXiv preprint arXiv: 1911.05755. vol. 73, No. 2. 2019, pp. 130-144. [cited by applicant]
Shiryaev, A.N. Probability. Springer, Second Edition. 1980, 22 pages. [cited by applicant]
Shorack et al. Empirical Processes with Applications to Statistics. Wiley, New York. 1986, 36 pages. [cited by applicant]
Villani, Cedric. Optimal Transport, Old and New. Springer. Jun. 13, 2008, 998 pages. [cited by applicant]
Woodworth et al. Learning Non-Discriminatory Predictors. Proceedings of Machine Learning Research, vol. 65, No. 1-34. 2017, 34 pages. [cited by applicant]
Zemel et al. Learning Fair Representations. Proceedings of the 30th International Conference on Machine Learning, PMLR vol. 28, No. 3. 2013, pp. 325-333. [cited by applicant]
Zhang et al. Mitigating Unwanted Biases with Adversarial Learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. Feb. 2-3, 2018, pp. 335-340. [cited by applicant]
Manu Joseph. Interpretability Cracking open the black box—Part 111. Machine Learning. Deep & Shallow, Nov. 24, 2019-Dec. 30, 2020, 29 pages. [cited by applicant]
Krivoruchko et al. Using Multivariate Interpolation for Estimating Well Performance, Esri.com, Arcuser. Esri Technology Summer 2014, 13 pages. URL:<URL: https://www.esri.com/abouUnewsroom/arcuser/using-multivariate-inte… [cited by applicant]
Chen et al. Comparison and Improvement of the predictability and interpretability with ensemble learning models in QSPR applications. Journal of Cheminformatics, Mar. 30, 2020, 50 pages. <URL:https://jcheminf.biomedcent… [cited by applicant]
Ding et al. Comparisons of Two Ensemble Mean Methods in Measuring the Average Error Growth and the Predictability. The Chinese Meteorological Society and Springer-Verlag Berlin Heidelberg, No. 4, vol. 25, Mar. 21, 2011,… [cited by applicant]
Ji et al. Post-Radiotherapy PET Image Outcome Prediction by Deep Learning Under Biological Model Guidance: A Feasibility Study of Oropharyngeal Cancer Application. Duke University Medical Center. arXiv preprint. 2021, 2… [cited by applicant]
Equal Credit Opportunity Act (ECOA). FDIC Law, Regulations, Related Actions, 9 pages [online]. Retrieved online: URL< https://www.fdic.gov/regulations/laws/rules/6000-1200.html>. [cited by applicant]
Thorpe, Matthew, Introduction to Optimal Transport, University of Cambridge (2018). [cited by applicant]
Miroshinikov et al. Mutual Information-based group explainers with coalition structure for machine learning model explanations. Computer Science and Game Theory. arXiv:2102.10878v1. Feb. 22, 2021, 46 pages. [cited by applicant]
Miroshinikov et al. Wasserstein-based faireness interpretability framework for machine learning models. arXiv:2011.03156v1. Nov. 6, 2020, 34 pages. [cited by applicant]
Miroshinikov et al. Model-agnostic bias mitigation methods with regressor distribution control for Wasserstein-based fairness metrics. arXiv:2111.11259v1. Nov. 19, 2021, 29 pages. [cited by applicant]
Blum et al. Recovering from Biased Data: Can Fairness Constraints Improve Accuracy? Toyota Technological Institute at Chicago arXiv:1912.01094v1 [cs.LG] Dec. 4, 2019, 20 pages. [cited by applicant]
Varley et al. Fairness in Machine Learning with Tractable Models. School of Informatics, University of Edinburgh, UK arXiv:1905.07026v2 [cs.LG] Jan. 13, 2020, 26 pages. [cited by applicant]
Hickey et al. Fairness by Explicability and Adversarial SHAP Learning. Experian UK&I and EMEA DataLabs, London, UK arXiv:2003.05330v3 [cs.LG] Jun. 26, 2020, 17 pages. [cited by applicant]
Manu Joseph. Interpretability Cracking open the black box—Part 11. Machine Learning. Deep & Shallow, Nov. 16, 2019-Nov. 28, 2019, 23 pages. [cited by applicant]
Aas et al. Explaining Individual Predictions When Features are Dependent: More Accurate Approximations to Shapley Values. arXiv preprint arXiv: 1903.10464v3, Feb. 6, 2020, 28 pages. [cited by applicant]
Albizuri et al. On Coalitional Semivalues. Games and Economic Behavior 49, Nov. 29, 2002, 37 pages. [cited by applicant]
Amer et al. The modified Banzhaf value for games with coalition structure: an axiomatic characterization. Mathematical Social Sciences 43, (2002) 45-54, 10 pages. [cited by applicant]
Alonso-Meijide et al. Modification of the Banzhaf Value for Games with a Coalition Structure. Annals of Operations Research, Jan. 2002, 17 pages. [cited by applicant]
Aumann et al. Cooperative Games with Coalition Structure. International journal of Game Theory, vol. 3, Issue 4, Jul. 1974, pp. 217-237. [cited by applicant]
Banzhaf III. Weighted Voting Doesn't Work: A Mathematical Analysis. Rutgers Law Review, vol. 19, No. 2, 1965, 28 pages. [cited by applicant]
Breiman et al. Estimating Optimal Transformations for Multiple Regression and Correlation. Journal of the American Statistical Association. vol. 80, No. 391, Sep. 1985, 19 pages. [cited by applicant]
Leo Breiman. Statistical Modeling: The Two Cultures. Statistical Science, vol. 16, No. 3, 2001, pp. 199-231. [cited by applicant]
Casas-Mendez et al. An extension of the tau-value to games with coalition structures. European Journal of Operational Research, vol. 148, 2003, pp. 494-513. [cited by applicant]
Chen et al. True to Model or True to the Data? arXiv preprint arXiv:2006.1623v1,Jun. 29, 2020, 7 pages. [cited by applicant]
Cover et al. Elements of Information Theory. A John Wiley & Sons, Inc., Publication. Second Edition, 2006, 774 pages. [cited by applicant]
Hall et al. An Introduction to Machine Learning Interpretability, O'Reilly, Second Edition, Aug. 2019, 62 pages. [cited by applicant]
Dickerson et al. Machine Learning. Considerations for fairly and transparently expanding access to credit. 2020, 29 pages [online]. Retrieved online: URL<https://info.h2o.ai/rs/644-PKX-778/images/Machine%20Learning%20-%… [cited by applicant]
Dubey et al. Value Theory Without Efficiency. Mathematics of Operations Research. Volume 6, No. 1. 1981, pp. 122-128. [cited by applicant]
Elshawi et al. On the interpretability of machine learning-based model for predicting hypertension. BMC Medical Informatics and Decision Making. vol. 19, No. 146. 2019, 32 pages. [cited by applicant]
Gretton et al. Measuring Statistical Dependence with Hilbert-Schmidt Norms. In Algorithmic Learning Theory. Springer-Verlag Berlin Heidelberg. 2005, pp. 63-77. [cited by applicant]
Gretton et al. A Kernel Statistical Test of Independence. In Advances in Neural Information Processing Systems. 2007, 8 pages. [cited by applicant]
Gretton et al. A Kernel Two-Sample Test. The Journal of Machine Learning Research. vol. 13, No. 1. Mar. 2012, pp. 723-773. [cited by applicant]
Hall, Patrick. On the Art and Science of Explainable Machine Learning: Techniques, Recommendations, and Responsibilties. KDD '19 XAI Workshop. arXiv preprint arxiv: 1810.02909, Aug. 2, 2019, 10 pages. [cited by applicant]
Heller et al. A consistent multivariate test of association based on ranks of distances. Biometrika. arXiv:1201.3522v1. Jan. 17, 2012, 14 pages. [cited by applicant]
Heller et al. Consistent Distribution-Free K-Sample and Independence Tests for Univariate Random Variables. Journal of Machine Learning Research. vol. 17, No. 29. Feb. 2016, 54 pages. [cited by applicant]
Hoeffding, Wassily. A Non-Parametric Test of Independence. The Annals of Mathematical Statistics. 1948, pp. 546-557. [cited by applicant]
Janzing et al. Feature relevance quantification in explainable AI: A causal problem. arXiv preprint arXiv:1910.13413v2. Nov. 25, 2019, 11 pages. [cited by applicant]
Jiang et al. Nonparametric K-Sample Tests via Dynamic Slicing. Journal of the American Statistical Association. Harvard Library. Sep. 11, 2015. pp. 642-653. [cited by applicant]
Jollife et al. Principal Component Analysis. Springer Series in Statistics. Second Edition, 2002, 518 pages. [cited by applicant]
Kamijo, Yoshio. A two-step Shapley value in a cooperative game with a coalition structure. International Game Theory Review. 2009, 11 pages. [cited by applicant]
Kraskov et al. Estimating mutual information. The American Physical Society. Physical Review E vol. 69. Jun. 23, 2004, 16 pages. [cited by applicant]
Lundberg, Scott. Welcome to SHAP Documentation. 2018, 46 pages [online]. Retrieved online URL: < https://shap.readthedocs.io/en/latest/>. [cited by applicant]
Lipovetsky et al. Analysis of regression in game theory approach. Applied Stochastic Models Business and Industry. vol. 17. Apr. 26, 2001, pp. 319-330. [cited by applicant]
Lopez-Paz et al. The Randomized Dependence Coefficient. In Advances in Neural Information Processing Systems. 2013, pp. 1-9. [cited by applicant]
Lorenzo-Freire, Silvia. New characterizations of the Owen and Banzhaf-Owen values using the intracoalitional balanced contributions property. Department of Mathematics. 2017, 23 pages. [cited by applicant]
Modarres et al. Towards Explainable Deep Learning for Credit Lending: A Case Study. arxiv: 1811.06471v2, Nov. 11, 2018, 8 pages. [cited by applicant]
Owen, Guillermo. Values of Games with a Priori Unions. In: Essays in Mathematical Economics and Game Theory. Springer, 1977, pp. 76-88. [cited by applicant]
G. Owen. Modification of the Banzhaf-Coleman index for Games with a Priori Unions. Power, Voting and Voting Power. 1981, pp. 232-238. [cited by applicant]
Paninski, Liam. Estimation of Entropy and Mutual Information. Neural Computation, vol. 15, No. 6. 2003, pp. 1191-1253. [cited by applicant]
Reshef et al. Detecting Novel Associations in Large Data Sets. Science, vol. 334. Dec. 2011, pp. 1518-1524. [cited by applicant]
Reshef et al. An Empirical Study of Leading Measures of Dependence. arXiv: 1505.02214v2. May 12, 2015, 42 pages. [cited by applicant]
Reshef et al. Measuring Dependence Powerfully and Equitably. Journal of Machine Learning Research, vol. 17. 2016, 63 pages. [cited by applicant]
Renyi. On Measures of Dependence. Acta mathematica hungarica, vol. 10, No. 3.1959, pp. 441-451. [cited by applicant]
Strumbelji et al. Explaining prediction models and individual predictions with feature contributions. Springer. Knowledge Information System, vol. 41, No. 3. 2014, pp. 647-665. [cited by applicant]
Sundararajan et al.The Many Shapley Values for Model Explanation. arXiv preprint arXiv:1908.08474v1, Aug. 22, 2019, 9 pages. [cited by applicant]
Szekely et al. Measuring and Testing Dependence by Correlation of Distances. The Annals of Statistics, vol. 35, No. 6. 2007. 26 pages. [cited by applicant]
Szekely et al. Brownian Distance Covariance. The Annals of Applied Statistics, vol. 3, No. 4. 2009. 31 pages. [cited by applicant]
Young. Monotonic Solutions of Cooperative Games International Journal of Game Theory, vol. 14, Issue 2. 1985. 8 pages. [cited by applicant]
Tibshirani et al. Estimating the number of clusters in a dataset via the gap statistic. Journal of the Royal Statistical Society, Series B. vol. 63, Part 2. 2001. pp. 411-423. [cited by applicant]
Tijs. Bounds for the Core and The Tau-Value. Game Theory and Mathematical Economics. 1981, pp. 123-132. [cited by applicant]
Vidal-Puga, Juan. The Harsanyi paradox and the ““right to talk”” in bargaining among coalitions. Mathematical Social Sciences. vol. 64. Mar. 27, 2012. 32 pages. [cited by applicant]
Zeng et al. Jackknife approach to the estimation of mutual information. PNAS, vol. 115, No. 40. Oct. 18, 2018. 6 pages. [cited by applicant]
Zhao et al. Causal Interpretations of Black-Box Models. J Bus., Econ. Stat., DOI:10.1080/07350015.2019.1624293, 2019, 16 pages. [cited by applicant]
Wang et al. Shapley Flow: A Graph-based Approach to Interpreting Model Predictions arXiv preprint arXiv:2010.14592, 2020, 11 pages. [cited by applicant]
U.S. Appl. No. 16/868,019, filed May 6, 2020. [cited by applicant]
Abduljabbar et al., “Applications of Artificial Intelligence in Transport: An Overview,” [cited by applicant]
Apley et al., “Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models,” [cited by applicant]
Aurenhammer et al., [cited by applicant]
Bzdok et al., “Statistics Versus Machine Learning,” Nature Methods, vol. 15, 233-234 (Apr. 2018). [cited by applicant]
Eggleston, H.G., [cited by applicant]
“Explainable Artificial Intelligence (xAI),” [cited by applicant]
FDIC Title VII—Equal Credit Opportunity, FDIC Law, Regulations, Related Acts—Consumer Protection (Dec. 31, 2019). [cited by applicant]
Friedman, J.H., “Greedy Function Approximation: A Gradient Boosting Machine,” [cited by applicant]
Goldstein et al., “Peeking Inside the Black Box: Visualizing Statistical Learning with Plots of Individual Conditional Expectation,” [cited by applicant]
Hall, Patrick, “On the Art and Science of Explainable Machine Learning: Techniques, Recommendations, and Responsibilities,” KDD '19 XAI Workshop, Aug. 4-8, 2019, Anchorage, AK. [cited by applicant]
Hastie et al., [cited by applicant]
Hu et al., “Locally Interpretable Models and Effects Based on Supervised Partitioning (LIME-SUP),” [cited by applicant]
Leo et al., “Machine Learning in Banking Risk Management: A Literature Review,” [cited by applicant]
Lundberg et al., “A Unified Approach to Interpreting Model Predictions,” [cited by applicant]
Lundberg et al., “Consistent Individualized Feature Attribution for Tree Ensembles,” ArXiv1802.03888 (2019). [cited by applicant]
Ribeiro et al., “‘Why Should I Trust You?’ Explaining the Predictions of Any Classifier,” [cited by applicant]
SAS/STAT® User's Guide, Version 8, Chapter 68 “The VARCLUS Procedure” (1999). [cited by applicant]
Shapley, L.S., “A Value for n-Person Games,” [cited by applicant]
Shiryaev, A.N., [cited by applicant]
Vaughan et al., “Explainable Neural Networks based on Additive Index Models,” [cited by applicant]
Wehle et al., “Machine Learning, Deep Learning, and AI: What's the Difference?” Data Scientist Innovation Day (2017). [cited by applicant]
Wuest et al., “Machine Learning in Manufacturing: Advantages, Challenges and Applications,” Production and Manufacturing Research, vol. 4, 23-45 (2016). [cited by applicant]
Vanderbei, Robert J. [cited by applicant]
Wasserman, Larry, [cited by applicant]
Bellamy et al. AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias, IBM J. Res. & Dev. vol. 63, No. 4/5, Paper 4. Jul./Sep. 2019. 15 pages. [cited by applicant]
Lohia et al. Bias Mitigation Post-Processing for Individual and Group Fairness. IBM Research and IBM Watson AI Platform. 978-1-5386-4658-8/18/$31.00. 2019. IEEE. 5 pages. [cited by applicant]