IP Library › Granted Patent US 12,614,083
Granted Patent B2
US 12,614,083 · App. 18/111,826 · Granted Apr 28, 2026

Computing system and method for applying Monte Carlo estimation to determine the contribution of individual input variables within dependent variable groups on the output of a data science model

Inventors: Konstandinos Kotsiopoulos (Easthampton, MA); Alexey Miroshnikov (Evanston, IL); Arjun Ravi Kannan (Buffalo Grove, IL)
Assignee: Capital One Financial Corporation
G06N5/022
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,614,083
App. No.
18/111,826
Granted
Apr 28, 2026
Kind
B2
Abstract

A computing platform is configured to, for a model object trained to output a score for an input data record, the input data record including a set of input variables that are arranged into groups based on dependencies, iterate the following for each respective input variable in each respective group: (a) identify a sample from a set of historical data records, (b) determine a first score for the sample, (c) determine a second score for the sample that is conditioned on the respective group in a given input data record, (d) select a random variable coalition within the respective group, (e) compute a group-specific contribution value for the respective input variable in the respective group, and (f) compute an iteration-specific contribution value for the respective input variable to the model output, and (vi) for each respective input variable, aggregate the iteration-specific contribution values and thereby determine an aggregated contribution value for the respective input variable.

Claims (99)

1 . A computing platform comprising:

at least one processor;

non-transitory computer-readable medium; and

program instructions stored on the non-transitory computer-readable medium that are executable by the at least one processor such that the computing platform is configured to:

train a model object for a data science model using a machine learning process, wherein the model object is trained to (i) receive an input data record comprising a set of input variables and (ii) output a score for the input data record;

arrange the set of input variables into two or more variable groups based on dependencies between respective input variables, where each variable group comprises at least one input variable;

identify a given input data record to be scored by the model object;

for each respective variable group of the model object, determine a contribution value for the respective variable group;

for each respective input variable in each respective variable group of the model object, perform a given number of iterations of:

identify a sample historical data record from a set of historical data records;

determine a first model object output score for the sample historical data record;

use the given input data record to determine a second model object output score for the sample historical data record that is conditioned on the respective variable group in the given input data record;

select a random variable coalition, within the respective variable group, comprising the respective input variable and zero or more other input variables;

use the given input data record, the sample historical data record, and the randomly-selected variable coalition to compute a group-specific contribution value for the respective input variable in the respective variable group; and

use (i) the contribution value for the respective variable group, (ii) the first model object output score, (iii) the second model object output score, and (iv) the group-specific contribution value for the respective input variable to compute an iteration-specific contribution value for the respective input variable to the model output; and

for each respective input variable of the model object, aggregate the iteration-specific contribution values calculated for each iteration and thereby determine an aggregated contribution value for the respective input variable.

2 . The computing platform of claim 1 , wherein the program instructions that are executable by the at least one processor such that the computing platform is configured to select the random variable coalition, within the respective variable group, comprising the respective input variable and zero or more other input variables comprise program instructions that are executable by the at least one processor such that the computing platform is configured to:

generate a random ordering of two or more input variables within the respective variable group; and

define the random variable coalition as including the respective input variable and all other input variables that precede the respective input variable in the generated random ordering.

3 . The computing platform of claim 1 , wherein the program instructions that are executable by the at least one processor such that the computing platform is configured to use the given input data record, the sample historical data record, and the randomly-selected variable coalition to compute the group-specific contribution value for the respective input variable in the respective group comprise program instructions that are executable by the at least one processor such that the computing platform is configured to:

generate a first synthetic data record comprising a mix of input variables from (i) the given input data record and (ii) the sample historical data record;

generate a second synthetic data record comprising an adjusted mix of input variables from (i) the given input data record and (ii) the sample historical data record;

use the trained model object to determine a first score for the first synthetic data record and a second score for the second synthetic data record; and

calculate a difference between the first score and the second score, wherein the difference is the group-specific contribution value.

4 . The computing platform of claim 3 , wherein the program instructions that are executable by the at least one processor such that the computing platform is configured to generate the first synthetic data record comprising the mix of input variables comprise program instructions that are executable by the at least one processor such that the computing platform is configured to:

identify a subset of input variables that are included in the randomly-selected variable coalition;

for the identified subset of input variables, use values from the given input data record for the first synthetic data record; and

for each other input variable, use values from the sample historical data record for the first synthetic data record; and

wherein the program instructions that are executable by the at least one processor such that the computing platform is configured to generate the second synthetic data record comprising the adjusted mix of input variables comprise program instructions that are executable by the at least one processor such that the computing platform is configured to:

for the identified subset of input variables, excluding the respective input variable, use values from the given input data record for the second synthetic data record; and

for each other input variable, including the respective input variable, use values from the sample historical data record for the second synthetic data record.

5 . The computing platform of claim 1 , wherein the program instructions that are executable by the at least one processor such that the computing platform is configured to aggregate the iteration-specific contribution values calculated for each iteration comprise program instructions that are executable by the at least one processor such that the computing platform is configured to determine an average of the iteration-specific contribution values for each iteration over the given number of iterations.

6 . The computing platform of claim 1 , wherein the program instructions that are executable by the at least one processor such that the computing platform is configured to, for each respective input variable in each respective variable group of the model object, perform the given number of iterations comprise program instructions that are executable by the at least one processor such that the computing platform is configured to:

while performing the given number of iterations for a first input variable of the model object, perform the given number of iterations for each other input variable of the model object.

7 . The computing platform of claim 1 , wherein the given number of iterations is 1,000 or more iterations.

8 . The computing platform of claim 1 , wherein the program instructions that are executable by the at least one processor such that the computing platform is configured to, for each respective input variable in each respective variable group of the model object, identify the sample historical data record from the set of historical data records comprise program instructions that are executable by the at least one processor such that the computing platform is configured to:

identify a randomly-sampled historical data record from the set of historical data records.

9 . A non-transitory computer-readable medium, wherein the non-transitory computer-readable medium is provisioned with program instructions that, when executed by at least one processor, cause a computing platform to:

train a model object for a data science model using a machine learning process, wherein the model object is trained to (i) receive an input data record comprising a set of input variables and (ii) output a score for the input data record;

arrange the set of input variables into two or more variable groups based on dependencies between respective input variables, where each variable group comprises at least one input variable;

identify a given input data record to be scored by the model object;

for each respective variable group of the model object, determine a contribution value for the respective variable group;

for each respective input variable in each respective variable group of the model object, perform a given number of iterations of:

identify a sample historical data record from a set of historical data records;

determine a first model object output score for the sample historical data record;

use the given input data record to determine a second model object output score for the sample historical data record that is conditioned on the respective variable group in the given input data record;

select a random variable coalition, within the respective variable group, comprising the respective input variable and zero or more other input variables;

use the given input data record, the sample historical data record, and the randomly-selected variable coalition to compute a group-specific contribution value for the respective input variable in the respective variable group; and

use (i) the contribution value for the respective variable group, (ii) the first model object output score, (iii) the second model object output score, and (iv) the group-specific contribution value for the respective input variable to compute an iteration-specific contribution value for the respective input variable to the model output; and

for each respective input variable of the model object, aggregate the iteration-specific contribution values calculated for each iteration and thereby determine an aggregated contribution value for the respective input variable.

10 . The non-transitory computer-readable medium of claim 9 , wherein the program instructions that, when executed by at least one processor, cause the computing platform to select the random variable coalition, within the respective variable group, comprising the respective input variable and zero or more other input variables comprise program instructions that, when executed by at least one processor, cause the computing platform to:

generate a random ordering of two or more input variables within the respective variable group; and

define the random variable coalition as including the respective input variable and all other input variables that precede the respective input variable in the generated random ordering.

11 . The non-transitory computer-readable medium of claim 9 , wherein the program instructions that, when executed by at least one processor, cause the computing platform to use the given input data record, the sample historical data record, and the randomly-selected variable coalition to compute the group-specific contribution value for the respective input variable in the respective group comprise program instructions that, when executed by at least one processor, cause the computing platform to:

generate a first synthetic data record comprising a mix of input variables from (i) the given input data record and (ii) the sample historical data record;

generate a second synthetic data record comprising an adjusted mix of input variables from (i) the given input data record and (ii) the sample historical data record;

use the trained model object to determine a first score for the first synthetic data record and a second score for the second synthetic data record; and

calculate a difference between the first score and the second score, wherein the difference is the group-specific contribution value.

12 . The non-transitory computer-readable medium of claim 11 , wherein the program instructions that, when executed by at least one processor, cause the computing platform to generate the first synthetic data record comprising the mix of input variables comprise program instructions that, when executed by at least one processor, cause the computing platform to:

identify a subset of input variables that are included in the randomly-selected variable coalition;

for the identified subset of input variables, use values from the given input data record for the first synthetic data record; and

for each other input variable, use values from the sample historical data record for the first synthetic data record; and

wherein the program instructions that, when executed by at least one processor, cause the computing platform to generate the second synthetic data record comprising the adjusted mix of input variables comprise program instructions that, when executed by at least one processor, cause the computing platform to:

for the identified subset of input variables, excluding the respective input variable, use values from the given input data record for the second synthetic data record; and

for each other input variable, including the respective input variable, use values from the sample historical data record for the second synthetic data record.

13 . The non-transitory computer-readable medium of claim 9 , wherein the program instructions that, when executed by at least one processor, cause the computing platform to aggregate the iteration-specific contribution values calculated for each iteration comprise program instructions that, when executed by at least one processor, cause the computing platform to determine an average of the iteration-specific contribution values for each iteration over the given number of iterations.

14 . The non-transitory computer-readable medium of claim 9 , wherein the program instructions that, when executed by at least one processor, cause the computing platform to, for each respective variable group of the model object, perform the given number of iterations comprise program instructions that, when executed by at least one processor, cause the computing platform to:

while performing the given number of iterations for a first variable group of the model object, perform the given number of iterations for each other variable group of the model object.

15 . The non-transitory computer-readable medium of claim 9 , wherein the given number of iterations is 1,000 or more iterations.

16 . The non-transitory computer-readable medium of claim 9 , wherein the program instructions that, when executed by at least one processor, cause the computing platform to, for each respective input variable in each respective variable group of the model object, identify the sample historical data record from the set of historical data records comprise program instructions that, when executed by at least one processor, cause the computing platform to:

identify a randomly-sampled historical data record from the set of historical data records.

17 . A method carried out by a computing platform, the method comprising:

training a model object for a data science model using a machine learning process, wherein the model object is trained to (i) receive an input data record comprising a set of input variables and (ii) output a score for the input data record;

arranging the set of input variables into two or more variable groups based on dependencies between respective input variables, where each variable group comprises at least one input variable;

identifying a given input data record to be scored by the model object;

for each respective variable group of the model object, determining a contribution value for the respective variable group;

for each respective input variable in each respective variable group of the model object, performing a given number of iterations of:

identifying a sample historical data record from a set of historical data records;

determining a first model object output score for the sample historical data record;

using the given input data record to determine a second model object output score for the sample historical data record that is conditioned on the respective variable group in the given input data record;

selecting a random variable coalition, within the respective variable group, comprising the respective input variable and zero or more other input variables;

using the given input data record, the sample historical data record, and the randomly-selected variable coalition to compute a group-specific contribution value for the respective input variable in the respective variable group; and

using (i) the contribution value for the respective variable group, (ii) the first model object output score, (iii) the second model object output score, and (iv) the group-specific contribution value for the respective input variable to compute an iteration-specific contribution value for the respective input variable to the model output; and

for each respective input variable of the model object, aggregating the iteration-specific contribution values calculated for each iteration and thereby determine an aggregated contribution value for the respective input variable.

18 . The method of claim 17 , wherein selecting the random variable coalition comprising the respective input variable and zero or more other input variables comprises:

generating a random ordering of two or more input variables within the respective variable group; and

defining the random variable coalition as including the respective input variable and all other input variables that precede the respective input variable in the generated random ordering.

19 . The method of claim 17 , wherein using the given input data record, the sample historical data record, and the randomly-selected variable coalition to compute the group-specific contribution value comprises:

generating a first synthetic data record comprising a mix of input variables from (i) the given input data record and (ii) the sample historical data record;

generating a second synthetic data record comprising an adjusted mix of input variables from (i) the given input data record and (ii) the sample historical data record;

using the trained model object to determine a first score for the first synthetic data record and a second score for the second synthetic data record; and

calculating a difference between the first score and the second score, wherein the difference is the group-specific contribution value.

20 . The method of claim 19 , wherein generating the first synthetic data record comprising the mix of input variables comprises:

identifying a subset of input variables that are included in the randomly-selected variable coalition;

for the identified subset of input variables, using values from the given input data record for the first synthetic data record; and

for each other input variable, using values from the sample historical data record for the first synthetic data record; and

wherein generating the second synthetic data record comprising the adjusted mix of input variables comprises:

for the identified subset of input variables, excluding the respective input variable, using values from the given input data record for the second synthetic data record; and

for each other input variable, including the respective input variable, using values from the sample historical data record for the second synthetic data record.

Assignments (2)
MERGER Recorded Jul 2, 2025
From: DISCOVER FINANCIAL SERVICES
To: CAPITAL ONE FINANCIAL CORPORATION
Reel/Frame 071784/0903 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 13, 2023
From: KOTSIOPOULOS, KONSTANDINOS; MIROSHNIKOV, ALEXEY; KANNAN, ARJUN RAVI
To: DISCOVER FINANCIAL SERVICES
Reel/Frame 062961/0778 →
Continuity (1)
Related Publication 20240281670A1 · Aug 22, 2024
References Cited (179)
US 8370253B1 · Grossman · 2013 [cited by applicant]
US 11030583B1 · Garg et al. · 2021 [cited by applicant]
US 11636386B2 · Lohia et al. · 2023 [cited by applicant]
US 12020133B2 · Goldszmidt et al. · 2024 [cited by applicant]
US 12277455B2 · Walters et al. · 2025 [cited by applicant]
US 20110270782A1 · Trenner et al. · 2011 [cited by applicant]
US 20140229415A1 · Martineau et al. · 2014 [cited by applicant]
US 20180285685A1 · Singh et al. · 2018 [cited by applicant]
US 20190087744A1 · Schiemenz · 2019 [cited by applicant]
US 20190188605A1 · Zavesky et al. · 2019 [cited by applicant]
US 20190279111A1 · Merrill et al. · 2019 [cited by applicant]
US 20190378210A1 · Merrill et al. · 2019 [cited by applicant]
US 20200081865A1 · Farrar et al. · 2020 [cited by applicant]
US 20200082299A1 · Affonso et al. · 2020 [cited by applicant]
US 20200184350A1 · Bhide et al. · 2020 [cited by applicant]
US 20200218987A1 · Jiang et al. · 2020 [cited by applicant]
US 20200302309A1 · Golding · 2020 [cited by applicant]
US 20200302524A1 · Kamkar · 2020 [cited by examiner]
US 20200311486A1 · Dey et al. · 2020 [cited by applicant]
US 20200320428A1 · Chaloulos et al. · 2020 [cited by applicant]
US 20200372035A1 · Tristan et al. · 2020 [cited by applicant]
US 20200372304A1 · Kenthapadi et al. · 2020 [cited by applicant]
US 20200372406A1 · Wick · 2020 [cited by applicant]
US 20200380398A1 · Weirder et al. · 2020 [cited by applicant]
US 20200410129A1 · Nadler et al. · 2020 [cited by applicant]
US 20210081828A1 · Lopez De Prado · 2021 [cited by applicant]
US 20210133870A1 · Kamkar et al. · 2021 [cited by applicant]
US 20210158102A1 · Lohia et al. · 2021 [cited by applicant]
US 20210173402A1 · Chang · 2021 [cited by examiner]
US 20210174222A1 · Dodwell et al. · 2021 [cited by applicant]
US 20210224605A1 · Zhang et al. · 2021 [cited by applicant]
US 20210224687A1 · Goldszmidt et al. · 2021 [cited by applicant]
US 20210241033A1 · Yang · 2021 [cited by applicant]
US 20210248503A1 · Hickety et al. · 2021 [cited by applicant]
US 20210256832A1 · Weisz et al. · 2021 [cited by applicant]
US 20210304039A1 · Tang · 2021 [cited by examiner]
US 20210334654A1 · Himanshi et al. · 2021 [cited by applicant]
US 20210350272A1 · Miroshnikov et al. · 2021 [cited by applicant]
US 20210383268A1 · Miroshnikov et al. · 2021 [cited by applicant]
US 20210383275A1 · Miroshnikov et al. · 2021 [cited by applicant]
US 20220036203A1 · Nachum et al. · 2022 [cited by applicant]
US 20240095605A1 · Fylstra · 2024 [cited by examiner]
WO 2021074491A1 · 2021 [cited by applicant]
W3Schools, Machine Learning—Train/Test, 2023 (Year: 2023). [cited by examiner]
Aas et al. Explaining Individual Predictions When Features are Dependent: More Accurate Approximations to Shapley Values. arXiv preprint arXiv:1903.10464v3. Feb. 6, 2020. 28 pages. [cited by applicant]
Abduljabbar et al. Applications of Artificial Intelligence in Transport: An Overview. Sustainability. 2019. 24 pages. [cited by applicant]
Casalicchio, Giuseppe. Visualizing the Feature Importance for Black Box Models. Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2018. Lecture Notes in Computer Science. vol. 11051. Springer. pp. 655-670… [cited by applicant]
Kwak, Nojun et al. Input Feature Selection for Classification Problems. IEEE Transactions on Neural Networks. vol. 13, No. 1. pp. 143-159. 2002. 17 pages. [cited by applicant]
Lundberg, Scott M., et al. Consistent Individualized Feature Attribution for Tree Ensembles. arXiv preprint arxiv:1802.03888v2. Jun. 18, 2018. 9 pages. [cited by applicant]
Wehle et al., “Machine Learning, Deep Learning, and AI: What's the Difference?” Data Scientist Innovation Day (2017). [cited by applicant]
Wuest et al., “Machine Learning in Manufacturing: Advantages, Challenges and Applications,” Production and Manufacturing Research, vol. 4, 23-45 (2016). [cited by applicant]
Zeng et al. Jackknife approach to the estimation of mutual information. PNAS, vol. 115, No. 40. Oct. 18, 2018. 6 pages. [cited by applicant]
Zhao et al. Causal Interpretations of Black-Box Models. J Bus., Econ. Stat., DOI:10.1080/07350015.2019.1624293, 2019, 16 pages. [cited by applicant]
International Search Report and Written Opinion issued in International Application No. PCT/US2022/029670, mailed on Sep. 7, 2022, 10 pages. [cited by applicant]
Perrone et al. Fair Bayesian Optimization. AIES '21, May 19-21, 2021, Virtual Event, USA. pp. 854-863. [cited by applicant]
Rawls, John. Justice as Fairness: A Restatement. Harvard University Press. 2001, 32 pages. [cited by applicant]
Reshef et al. An Empirical Study of Leading Measures of Dependence. arXiv preprint arXiv:1505.02214. May 12, 2015, 42 pages. [cited by applicant]
Reshef et al. Detecting Novel Associations in Large Data Sets. Science, vol. 334. Dec. 2011, pp. 1518-1524. [cited by applicant]
Reshef et al. Measuring Dependence Powerfully and Equitably. Journal of Machine Learning Research, vol. 17. 2016, 63 pages. [cited by applicant]
Ribeiro et al. “Why Should I Trust You?” Explaining the Predictions of Any Classifier. 22nd Conference on Knowledge Discovery and Data Mining, San Francisco, CA. arXiv:1602.04938v3. Aug. 9, 2016, 10 pages. [cited by applicant]
Royden et al. Real Analysis. Boston: Prentice Hall, 4th Edition. 2010, 516 pages. [cited by applicant]
Santambrogio, Filippo. Optimal Transport for Applied Mathematicians. Calculus of Variations, PDEs and Modeling. Springer. May 2015, 356 pages. [cited by applicant]
Schmidt et al. An Introduction to Artificial Intelligence and Solutions to the Problems of Algorithmic Discrimination. arXiv preprint arXiv:1911.05755. vol. 73, No. 2. 2019, pp. 130-144. [cited by applicant]
Shapley, L.S. A Value for n-Person Games. Annals of Mathematics Studies. 1953, 13 pages. [cited by applicant]
Shiryaev, A.N. Probability. Springer, Second Edition. 1980, 22 pages. [cited by applicant]
Shorack et al. Empirical Processes with Applications to Statistics. Wiley, New York. 1986, 36 pages. [cited by applicant]
Strumbelji et al. Explaining Prediction Models and Individual Predictions with Feature Contributions. Springer. Knowledge Information System, vol. 41, No. 3. 2014, pp. 647-665. [cited by applicant]
Sundararajan et al. The Many Shapley Values for Model Explanation. arXiv preprint arXiv:1908.08474v1, Aug. 22, 2019, 9 pages. [cited by applicant]
Szekely et al. Measuring and Testing Dependence by Correlation of Distances. The Annals of Statistics, vol. 35, No. 6. ArXiv:0803.4101v1, Mar. 28, 2008, 26 pages. [cited by applicant]
Thorpe, Matthew, Introduction to Optimal Transport, University of Cambridge, 2018, 56 pages. [cited by applicant]
Villani, Cedric. Optimal Transport, Old and New. Springer. Jun. 13, 2008, 998 pages. [cited by applicant]
Wasserman et al. All of Statistics: A Concise Course in Statistical Inference. Springer Texts in Statistics. 2004, 14 pages. [cited by applicant]
Woodworth et al. Learning Non-Discriminatory Predictors. Proceedings of Machine Learning Research, vol. 65, No. 1-34. 2017, 34 pages. [cited by applicant]
Young. Monotonic Solutions of Cooperative Games International Journal of Game Theory, vol. 14, Issue 2. 1985. 8 pages. [cited by applicant]
Zemel et al. Learning Fair Representations. Proceedings of the 30th International Conference on Machine Learning, PMLR vol. 28, No. 3. 2013, pp. 325-333. [cited by applicant]
Zhang et al. Mitigating Unwanted Biases with Adversarial Learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. Feb. 2-3, 2018, pp. 335-340. [cited by applicant]
Balashankar et al. Pareto-Efficient Fairness for Skewed Subgroup Data. In the International Conference on Machine earning AI for Social Good Workshop. vol. 8. Long Beach, United States, 2019, 8 pages. [cited by applicant]
Banzhaf III. Weighted Voting Doesn't Work: A Mathematical Analysis. Rutgers Law Review, vol. 19, No. 2, 1965, 28 pages. [cited by applicant]
Barocas et al. Fairness and Machine Learning: Limitations and Opportunities. [online], 253 pages, 1999 [retrieved online on May 26, 2022]. Retrieved from the Internet :<URL:https://fairmlbook.org/>. [cited by applicant]
Bergstra et al. Algorithms for Hyper-Parameter Optimization. NIPS 11: Proceedings of the 24th International conference on Neural Information Processing Systems. Dec. 2011, 9 pages. [cited by applicant]
Bousquet et al. Stability and Generalization. Journal of Machine Learning Research 2. Mar. 2002, pp. 499-526. [cited by applicant]
Chen et al. True to Model or True to the Data? arXiv preprint arXiv:2006.1623v1, Jun. 29, 2020, 7 pages. [cited by applicant]
Cover et al. Elements of Information Theory. A John Wiley & Sons, Inc., Publication. Second Edition, 2006, 774 pages. [cited by applicant]
Del Barrio et al. Obtaining Fairness Using Optimal Transport Theory. arXiv preprint arXiv:1806.03195v2. Jul. 19, 2018, 25 pages. [cited by applicant]
Del Barrio et al. On Approximate Validation of Models: A Kolmogorov-Smirnov Based Approach. arXiv:1903.08687v1, Mar. 20, 2019, 32 pages. [cited by applicant]
Dickerson et al. Machine Learning. Considerations for fairly and transparently expanding access to credit. 2020, 29 pages [online]. Retrieved online: URL<https://info.h2o.ai/rs/644-PKX-778/images/Machine%20Learning%20-%… [cited by applicant]
Dwork et al. Generalization in Adaptive Data Analysis and Holdout Reuse. NIPS'15: Proceedings of the 28th International Conference on Neural Information Processing Systems, vol. 2, Dec. 2015, 9 pages. [cited by applicant]
Dwork et al. Fairness Through Awareness. Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, 2012, 214-226. [cited by applicant]
Equal Credit Opportunity Act (ECOA). FDIC Law, Regulations, Related Actions, 9 pages [online]. Retrieved online: URL < https://www.fdic.gov/regulations/laws/rules/6000-1200.html>. [cited by applicant]
Equal Employment Opportunity Act. 2 pages [online]. Retrieved online: URL<https://www.dol.gov/sites/dolgov/files/ofccp/regs/compliance/posters/pdf/eeopost.pdf>. [cited by applicant]
Fair Housing Act (FHA). FDIC Law, Regulations, Related Actions, 2022, 3 pages [online]. Retrieved online: URL<https://www.ecfr.gov/current/title-12/chapter-III/subchapter-B/part-338. [cited by applicant]
Feldman et al. Certifying and Removing Disparate Impact. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015, 259-268. [cited by applicant]
Friedman, Jerome. 1999 Reitz Lecture. Greedy Function Approximation: A Gradient Boosting Machine. Annals of Statistics, vol. 29, No. 6. 2001, pp. 1189-1232. [cited by applicant]
G. Owen. Modification of the Banzhaf-Coleman index for Games with a Priori Unions. Power, Voting and Voting Power. 1981, pp. 232-238. [cited by applicant]
Goldstein et al. Peeking Inside the Black Box: Visualizing Statistical Learning with Plots of Individual Conditional Expectation. arXiv:1309.6392v2J. Mar. 20, 2014, 22 pages. [cited by applicant]
Hall et al. A United States Fair Lending Perspective on Machine Learning. Frontiers in Artificial Intelligence. doi: 10.3389/frai.2021.695301, Jun. 7, 2021, 9 pages. [cited by applicant]
Hall, Patrick. On the Art and Science of Explainable Machine Learning: Techniques, Recommendations, and Responsibilities. KDD'19 XAI Workshop. arXiv preprint arxiv:1810.02909, Aug. 2, 2019, 10 pages. [cited by applicant]
Hardt et al. Equality of Opportunity in Supervised Learning. Advances in Neural Information Processing Systems, 2015, 3315-3323. [cited by applicant]
Hashimoto et al. Fairness Without Demographics in Repeated Loss Minimization. In ICML, 2018, 10 pages. [cited by applicant]
Hastie et al. The Elements of Statistical Learning: Data Mining, Interference, and Prediction. Springer. Second edition, Jan. 13, 2017, 764 pages. [cited by applicant]
Heller et al. A consistent multivariate test of association based on ranks of distances. Biometrika. arXiv:1201.3522v1. Jan. 17, 2012, 14 pages. [cited by applicant]
Heller et al. Consistent Distribution-Free K-Sample and Independence Tests for Univariate Random Variables. Journal of Machine Learning Research. vol. 17, No. 29. Feb. 2016, 54 pages. [cited by applicant]
Janzing et al. Feature relevance quantification in explainable AI:A causal problem. arXiv preprint arXiv:1910.13413v2. Nov. 25, 2019, 11 pages. [cited by applicant]
Jiang et al. Identifying and Correcting Label Bias in Machine Learning. Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS). 2020, 10 pages. [cited by applicant]
Jiang et al. Smooth Isotonic Regression: A New Method to Calibrate Predictive Models. AMIA Joint Summits on Translational Science proceedings. 2011, pp. 16-20. [cited by applicant]
Kamiran et al. Classifying without Discriminating. 2009 2nd International Conference on Computer, Control and Communication. doi: 10.1109/IC4.2009.4909197. 2009, pp. 1-6. [cited by applicant]
Kamiran et al. Data Preprocessing Techniques for Classification Without Discrimination. Knowl. Inf. Syst. DOI 10.1007/s10115-011-0463-8. Dec. 3, 2011, 33 pages. [cited by applicant]
Kamiran et al. Discrimination Aware Decision Tree Learning. 2010 IEEE International Conference on Data Mining. doi: 10.1109/ICDM.2010.50. 2010, pp. 869-874. [cited by applicant]
Kamishima et al. Fairness-Aware Classifier with Prejudice Remover Regularizer. Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECMLPKDD), Part … [cited by applicant]
Karush, William. Minima of Functions of Several Variables with Inequalities as Side Constraints. A Dissertation Submitted to the Faculty of the Division of the Physical Sciences in Candidacy for the Degree of Master in … [cited by applicant]
Kearns et al. Algorithmic Stability and Sanity-Check Bounds for Leave-One-Out Cross-Validation. Neural Computation, vol. 11, No. 6. 1999, pp. 1427-1453. [cited by applicant]
Koralov et al. Theory of Probability and Random Processes. Second Edition. Springer. 1998, 349 pages. [cited by applicant]
Kovalev et al. A Robust Algorithm for Explaining Unreliable Machine Learning Survival Models Using the Kolmogorov-Smirnov Bounds. Neural Networks. arXiv:2005.02249v1, May 5, 2020. pp. 1-39. [cited by applicant]
Kuhn et al. Nonlinear programming. In Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability. Berkeley, Calif. University of California Press. 90. 1951, pp. 481-492. [cited by applicant]
Lahoti et al. Fairness without Demographics through Adversarially Reweighted Learning. arXiv preprint arXiv:2006.13114. 2020, 13 pages. [cited by applicant]
Lipovetsky et al. Analysis of Regression in Game Theory Approach. Applied Stochastic Models Business and Industry. vol. 17. Apr. 26, 2001, pp. 319-330. [cited by applicant]
Lorenzo-Freire, Silvia. New characterizations of the Owen and Banzhaf-Owen values using the intracoalitional balanced contributions property. Department of Mathematics. 2017, 23 pages. [cited by applicant]
Lundberg et al. Consistent Individualized Feature Attribution for Tree Ensembles. arXiv preprint arxiv:1802.03888v3. Mar. 7, 2019, 9 pages. [cited by applicant]
Lundberg et al. A Unified Approach to Interpreting Model Predictions. 31st Conference on Neural Information Processing Systems, 2017, 10 pages. [cited by applicant]
Markowitz, Harry. Portfolio Selection. The Journal of Finance. vol. 7, No. 1. 1952, pp. 77-91. [cited by applicant]
Miroshinikov et al. Model-Agnostic Bias Mitigation Methods with Regressor Distribution Control for Wasserstein-Based Fairness Metrics. arXiv:2111.11259v1. Nov. 19, 2021, 29 pages. [cited by applicant]
Miroshinikov et al. Mutual Information-Based Group Explainers with Coalition Structure for Machine Learning Model Explanations. Computer Science and Game Theory. arXiv:2102.10878v1. Feb. 22, 2021, 46 pages. [cited by applicant]
Miroshinikov et al. Wasserstein-Based Fairness Interpretability Framework for Machine Learning Models. arXiv:2011.03156v1. Nov. 6, 2020, 34 pages. [cited by applicant]
Owen, Guillermo. Values of Games with a Priori Unions. In: Essays in Mathematical Economics and Game Theory. Springer, 1977, pp. 76-88. [cited by applicant]
Pareto. Optimality. In: Multiobjective Linear Programming. Springer International Publishing. DOI 10.1007/978-3-319-21091-9_4. 2016, pp. 85-118. [cited by applicant]
Blum et al. Recovering from Biased Data: Can Fairness Constraints Improve Accuracy? Toyota Technological Institute at Chicago arXiv:1912.01094v1 [cs.LG] Dec. 4, 2019, 20 pages. [cited by applicant]
Varley et al. Fairness in Machine Learning with Tractable Models. School of Informatics, University of Edinburgh, UK arXiv:1905.07026v2 [cs.LG] Jan. 13, 2020, 26 pages. [cited by applicant]
Hickey et al. Fairness by Explicability and Adversarial SHAP Learning. Experian UK&I and EMEA DataLabs, London, UK arXiv:2003.05330v3 [cs.LG] Jun. 26, 2020, 17 pages. [cited by applicant]
Manu Joseph. Interpretability Cracking open the black box—Part 11. Machine Learning. Deep & Shallow, Nov. 16, 2019-Nov. 28, 2019, 23 pages. [cited by applicant]
Manu Joseph. Interpretability Cracking open the black box—Part 111. Machine Learning. Deep & Shallow, Nov. 24, 2019-Dec. 30, 2020, 29 pages. [cited by applicant]
Krivoruchko et al. Using Multivariate Interpolation for Estimating Well Performance, Esri.com, Arcuser. Esri Technology Summer 2014, 13 pages. URL:<URL: https://www.esri.com/abouUnewsroom/arcuser/using-multivariate-inte… [cited by applicant]
Chen et al. Comparison and Improvement of the predictability and interpretability with ensemble learning models in QSPR applications. Journal of Cheminformatics, Mar. 30, 2020, 50 pages. <URL:https://jcheminf.biomedcent… [cited by applicant]
Ding et al. Comparisons of Two Ensemble Mean Methods in Measuring the Average Error Growth and the Predictability. The Chinese Meteorological Society and Springer-Verlag Berlin Heidelberg, No. 4, vol. 25, Mar. 21, 2011,… [cited by applicant]
Hu et al. Locally Interpretable Models and Effects based on Supervised Partitioning, Wells Fargo, Jun. 1, 2018, 15 pages <URL: https://arxiv.org/pdf/1806.00663>. [cited by applicant]
Jiang et al. Wasserstein Fair Classification. Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, arXiv:1907.12059v1 [stat.ML] Jul. 28, 2019, 15 pages. <URL: chrome-extension://efaidnbm… [cited by applicant]
Wei et al. Optimized Score Transformation for Consistent Fair Classication, arXiv:1906.00066v3 [cs.LG] Oct. 29, 2021, 78 pages <URL: chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://arxiv.org/pdf/1906.00066.p… [cited by applicant]
Iosifidis et al. FAE: A Fairness-Aware Ensemble Framework, IEEE, arXiv:2002.00695v1 [cs.AI] Feb. 3, 2020, 6 pages. <URL:https://arxiv.org/abs/2002.00695>. [cited by applicant]
Abduljabbar et al., “Applications of Artificial Intelligence in Transport: An Overview,” Sustainability (2019). [cited by applicant]
Albizuri et al. On Coalitional Semivalues. Games and Economic Behavior 49, Nov. 29, 2002, 37 pages. [cited by applicant]
Alonso-Meijide et al. Modification of the Banzhaf Value for Games with a Coalition Structure. Annals of Operations Research, Jan. 2002, 17 pages. [cited by applicant]
Amer et al. The modified Banzhaf value for games with coalition structure: an axiomatic characterization. Mathematical Social Sciences 43, (2002) 45-54, 10 pages. [cited by applicant]
Apley et al., “Visualizing the Effects of Predictor Variables in Black Box Supervised Leaming Models,” Ar Xiv (2016). [cited by applicant]
Aumann et al. Cooperative Games with Coalition Structure. International journal of Game Theory, vol. 3, Issue 4, Jul. 1974, pp. 217-237. [cited by applicant]
Aurenhammer et al., Voronoi Diagrams and Delaunay Triangulations, pp. 7-13, 41-45, New Jersey, World Scientific Publishing (2013). [cited by applicant]
Breiman et al. Estimating Optimal Transformations for Multiple Regression and Correlation. Journal of the American Statistical Association. vol. 80, No. 391, Sep. 1985, 19 pages. [cited by applicant]
Bzdok et al., “Statistics Versus Machine Learning,” Nature Methods, vol. 15, 233-234 (Apr. 2018). [cited by applicant]
Casas-Mendez et al. An extension of the tau-value to games with coalition structures. European Journal of Operational Research, vol. 148, 2003, pp. 494-513. [cited by applicant]
Dubey et al. Value Theory Without Efficiency. Mathematics of Operations Research. vol. 6, No. 1. 1981, pp. 122-128. [cited by applicant]
Eggleston, H.G., Convexity, pp. 33-38, Cambridge University Press (1958). [cited by applicant]
Elshawi et al. On the interpretability of machine learning-based model for predicting hypertension. BMC Medical Informatics and Decision Making. vol. 19, No. 146. 2019, 32 pages. [cited by applicant]
Explainable Artificial Intelligence (xAI), Defense Advanced Research Projects Agency, DARPA-BAA-16-53 (2016). [cited by applicant]
FDIC Title VII-Equal Credit Opportunity, FDIC Law, Regulations, Related Acts—Consumer Protection (Dec. 31, 2019). [cited by applicant]
Gretton et al. A Kernel Statistical Test of Independence. In Advances in Neural Information Processing Systems. 2007, 8 pages. [cited by applicant]
Gretton et al. A Kernel Two-Sample Test. The Journal of Machine Learning Research. vol. 13, No. 1. Mar. 2012, pp. 723-773. [cited by applicant]
Gretton et al. Measuring Statistical Dependence with Hilbert-Schmidt Norms. In Algorithmic Learning Theory. Springer-Verlag Berlin Heidelberg. 2005, pp. 63-77. [cited by applicant]
Hall et al. An Introduction to Machine Learning Interpretability, O'Reilly, Second Edition, Aug. 2019, 62 pages. [cited by applicant]
Hoeffding, Wassily. A Non-Parametric Test of Independence. The Annals of Mathematical Statistics. 1948, pp. 546-557. [cited by applicant]
Ji et al. Post-Radiotherapy PET Image Outcome Prediction by Deep Learning Under Biological Model Guidance: A Feasibility Study of Oropharyngeal Cancer Application. Duke University Medical Center. arXiv preprint. 2021, 2… [cited by applicant]
Jiang et al. Nonparametric K-Sample Tests via Dynamic Slicing. Journal of the American Statistical Association. Harvard Library. Sep. 11, 2015. pp. 642-653. [cited by applicant]
Jollife et al. Principal Component Analysis. Springer Series in Statistics. Second Edition, 2002, 518 pages. [cited by applicant]
Kamijo, Yoshio. A two-step Shapley value in a cooperative game with a coalition structure. International Game Theory Review. 2009, 11 pages. [cited by applicant]
Kraskov et al. Estimating mutual information. The American Physical Society. Physical Review E vol. 69. Jun. 23, 2004, 16 pages. [cited by applicant]
Leo Breiman. Statistical Modeling: The Two Cultures. Statistical Science, vol. 16, No. 3, 2001, pp. 199-231. [cited by applicant]
Leo et al., “Machine Learning in Banking Risk Management: A Literature Review,” Risks (2019). [cited by applicant]
Lopez-Paz et al. The Randomized Dependence Coefficient. In Advances in Neural Information Processing Systems. 2013, pp. 1-9. [cited by applicant]
Lundberg, Scott. Welcome to SHAP Documentation. 2018, 46 pages [online]. Retrieved online URL :< https://shap.readthedocs.io/en/latest/>. [cited by applicant]
Miroshinikov et al. Wasserstein-based faireness interpretability framework for machine learning models. arXiv:2011.03156v1. Nov. 6, 2020, 34 pages. [cited by applicant]
Modarres et al. Towards Explainable Deep Learning for Credit Lending: A Case Study. arxiv:1811.06471v2, Nov. 11, 2018, 8 pages. [cited by applicant]
Paninski, Liam. Estimation of Entropy and Mutual Information. Neural Computation, vol. 15, No. 6. 2003, pp. 1191-1253. [cited by applicant]
Renyi. On Measures of Dependence. Acta mathematica hungarica, vol. 10, No. 3.1959, pp. 441-451. [cited by applicant]
SAS/STAT User's Guide, Version 8, Chapter 68 “The VARCLUS Procedure” (1999). [cited by applicant]
Shiryaev, A.N., Graduate Texts in Mathematics: Probability. 2nd ed., pp. 23-43, Springer (1980). [cited by applicant]
Szekely et al. Brownian Distance Covariance. The Annals of Applied Statistics, vol. 3, No. 4. 2009. 31 pages. [cited by applicant]
Tibshirani et al. Estimating the number of clusters in a dataset via the gap statistic. Journal of the Royal Statistical Society, Series B. vol. 63, Part 2. 2001. pp. 411-423. [cited by applicant]
Tijs. Bounds for the Core and the Tau-Value. Game Theory and Mathematical Economics. 1981, pp. 123-132. [cited by applicant]
Vanderbei, Robert J. Linear Programming: Foundations and Extensions, 4th ed., pp. 11-16, 141-144, New York, NY, Springer Science and Business Media (2014). [cited by applicant]
Vaughan et al., “Explainable Neural Networks based on Additive Index Models,” Corporate Model Risk, Wells Fargo, USA (Jun. 2018). [cited by applicant]
Vidal-Puga, Juan. The Harsanyi paradox and the “right to talk” in bargaining among coalitions. Mathematical Social Sciences. vol. 64. Mar. 27, 2012. 32 pages. [cited by applicant]
Wang et al. Shapley Flow. A Graph-based Approach to Interpreting Model Predictions arXiv preprint arXiv:2010.14592, 2020, 11 pages. [cited by applicant]