IP Library › Granted Patent US 12,657,398
Granted Patent B2
US 12,657,398 · App. 18/280,792 · Granted Jun 16, 2026

System, method, and computer program product for debiasing embedding vectors of machine learning models

Inventors: Sunipa Dev (Los Angeles, CA); Yan Zheng (Los Gatos, CA); Michael Yeh (Newark, CA); Junpeng Wang (Santa Clara, CA); Wei Zhang (Fremont, CA); Archit Rathore (Salt Lake City, UT)
Assignee: Visa International Service Association
G06F40/40
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,657,398
App. No.
18/280,792
Granted
Jun 16, 2026
Kind
B2
Abstract

Described are a system, method, and computer program product for debiasing embedding vectors of machine learning models. The method includes receiving embedding vectors and generating two clusters thereof. The method includes determining a first mean vector of the first cluster and a second mean vector of the second cluster. The method includes determining a bias associated with each of a plurality of first candidate vectors and replacing the first mean vector with a first candidate vector based on the bias. The method includes determining a bias associated with each of a plurality of second candidate vectors and replacing the second mean vector with a second candidate vector based on the bias. The method includes repeatedly replacing the first and second mean vectors until an extremum of the bias score is reached, and debiasing the embedding vectors by linear projection using a direction defined by the first and second mean vectors.

Claims (63)

1 . A computer-implemented method comprising:

receiving, with at least one processor, a plurality of embedding vectors from a neural network model;

determining, with the at least one processor, a bias vector by:

generating, with the at least one processor, two clusters of embedding vectors based on the plurality of embedding vectors, the two clusters comprising a first cluster of embedding vectors expected to be biased in a first direction and a second cluster expected to be biased in a second direction;

determining, with the at least one processor, a first mean vector of the first cluster of embedding vectors and a second mean vector of the second cluster of embedding vectors;

modifying, with the at least one processor, the first mean vector by moving the first mean vector toward each embedding vector of the first cluster of embedding vectors to provide a plurality of first candidate vectors and linearly projecting the embedding vectors of both the first cluster and the second cluster along a first direction defined between the second mean vector and each first candidate vector to determine a bias score associated with each first candidate vector;

replacing, with the at least one processor, the first mean vector with a first candidate vector of the plurality of first candidate vectors based on the bias score of the first candidate vector;

modifying, with the at least one processor, the second mean vector by moving the second mean vector toward each embedding vector of the second cluster of embedding vectors to provide a plurality of second candidate vectors and linearly projecting the embedding vectors of both the first cluster and the second cluster along a second direction defined between the first mean vector and each second candidate vector to determine the bias score associated with each second candidate vector;

replacing, with the at least one processor, the second mean vector with a second candidate vector of the plurality of second candidate vectors based on the bias score of the second candidate vector; and

repeating, with the at least one processor, the modifying of the first mean vector, the replacing of the first mean vector, the modifying of the second mean vector, and the replacing of the second mean vector until an extremum of the bias score is reached,

wherein the bias vector is based on a direction defined between the first mean vector and the second mean vector;

in response to reaching the extremum of the bias score, linearly projecting, with the at least one processor, each embedding vector of the plurality of embedding vectors along the bias vector to debias the plurality of embedding vectors to provide a plurality of debiased embedding vectors;

training, with at least one processor, a machine learning model of a fraud monitoring system based on the plurality of debiased embedding vectors;

receiving, with at least one processor, an authorization request associated with a transaction; and

declining, with at least one processor, the transaction associated with the authorization request based on the machine learning model of the fraud monitoring system as trained based on the plurality of debiased embedding vectors.

2 . The computer-implemented method of claim 1 , wherein the plurality of embedding vectors are vector representations of merchant identity embeddings generated from customer transaction data.

3 . The computer-implemented method of claim 1 , wherein the bias score is calculated from a Word Embedding Association Test (WEAT).

4 . The computer-implemented method of claim 1 , wherein the bias score is calculated from an Embedding Coherence Test (ECT).

5 . The computer-implemented method of claim 1 , further comprising, before modifying the first mean vector and the second mean vector, determining, with the at least one processor, an initial bias score by linearly projecting the embedding vectors of both the first cluster and the second cluster along an initial direction defined between the first mean vector and the second mean vector, wherein:

the replacing of the first mean vector with the first candidate vector of the plurality of first candidate vectors is based on maximizing a difference between the initial bias score and the bias score of the first candidate vector; and

the replacing of the second mean vector with the second candidate vector of the plurality of second candidate vectors is based on maximizing a difference between the initial bias score and the bias score of the second candidate vector.

6 . A system comprising at least one server comprising at least one processor, the at least one server programmed or configured to:

receive a plurality of embedding vectors from a neural network model;

determine a bias vector by:

generating two clusters of embedding vectors based on the plurality of embedding vectors, the two clusters comprising a first cluster of embedding vectors expected to be biased in a first direction and a second cluster expected to be biased in a second direction;

determining a first mean vector of the first cluster of embedding vectors and a second mean vector of the second cluster of embedding vectors;

modifying the first mean vector by moving the first mean vector toward each embedding vector of the first cluster of embedding vectors to provide a plurality of first candidate vectors and linearly projecting the embedding vectors of both the first cluster and the second cluster along a first direction defined between the second mean vector and each first candidate vector to determine a bias score associated with each first candidate vector;

replacing the first mean vector with a first candidate vector of the plurality of first candidate vectors based on the bias score of the first candidate vector;

modifying the second mean vector by moving the second mean vector toward each embedding vector of the second cluster of embedding vectors to provide a plurality of second candidate vectors and linearly projecting the embedding vectors of both the first cluster and the second cluster along a second direction defined between the first mean vector and each second candidate vector to determine the bias score associated with each second candidate vector;

replacing the second mean vector with a second candidate vector of the plurality of second candidate vectors based on the bias score of the second candidate vector; and

repeating the modifying of the first mean vector, the replacing of the first mean vector, the modifying of the second mean vector, and the replacing of the second mean vector until an extremum of the bias score is reached,

wherein the bias vector is based on a direction defined between the first mean vector and the second mean vector;

in response to reaching the extremum of the bias score, linearly project each embedding vector of the plurality of embedding vectors along the bias vector to debias the plurality of embedding vectors to provide a plurality of debiased embedding vectors;

train a machine learning model of a fraud monitoring system based on the plurality of debiased embedding vectors;

receive an authorization request associated with a transaction; and

decline the transaction associated with the authorization request based on the machine learning model of the fraud monitoring system as trained based on the plurality of debiased embedding vectors.

7 . The system of claim 6 , wherein the plurality of embedding vectors are vector representations of merchant identity embeddings generated from customer transaction data.

8 . The system of claim 6 , wherein the bias score is calculated from a Word Embedding Association Test (WEAT).

9 . The system of claim 6 , wherein the bias score is calculated from an Embedding Coherence Test (ECT).

10 . The system of claim 6 , wherein the at least one server is further programmed or configured to, before modifying the first mean vector and the second mean vector, determine an initial bias score by linearly projecting the embedding vectors of both the first cluster and the second cluster along an initial direction defined between the first mean vector and the second mean vector, and wherein:

the replacing of the first mean vector with the first candidate vector of the plurality of first candidate vectors is based on maximizing a difference between the initial bias score and the bias score of the first candidate vector; and

the replacing of the second mean vector with the second candidate vector of the plurality of second candidate vectors is based on maximizing a difference between the initial bias score and the bias score of the second candidate vector.

11 . A computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to:

receive a plurality of embedding vectors from a neural network model;

determine a bias vector by:

generating two clusters of embedding vectors based on the plurality of embedding vectors, the two clusters comprising a first cluster of embedding vectors expected to be biased in a first direction and a second cluster expected to be biased in a second direction;

determining a first mean vector of the first cluster of embedding vectors and a second mean vector of the second cluster of embedding vectors;

modifying the first mean vector by moving the first mean vector toward each embedding vector of the first cluster of embedding vectors to provide a plurality of first candidate vectors and linearly projecting the embedding vectors of both the first cluster and the second cluster along a first direction defined between the second mean vector and each first candidate vector to determine a bias score associated with each first candidate vector;

replacing the first mean vector with a first candidate vector of the plurality of first candidate vectors based on the bias score of the first candidate vector;

modifying the second mean vector by moving the second mean vector toward each embedding vector of the second cluster of embedding vectors to provide a plurality of second candidate vectors and linearly projecting the embedding vectors of both the first cluster and the second cluster along a second direction defined between the first mean vector and each second candidate vector to determine the bias score associated with each second candidate vector;

replacing the second mean vector with a second candidate vector of the plurality of second candidate vectors based on the bias score of the second candidate vector; and

repeating the modifying of the first mean vector, the replacing of the first mean vector, the modifying of the second mean vector, and the replacing of the second mean vector until an extremum of the bias score is reached,

wherein the bias vector is based on a direction defined between the first mean vector and the second mean vector;

in response to reaching the extremum of the bias score, linearly project each embedding vector of the plurality of embedding vectors along the bias vector to debias the plurality of embedding vectors to provide a plurality of debiased embedding vectors;

train a machine learning model of a fraud monitoring system based on the plurality of debiased embedding vectors;

receive an authorization request associated with a transaction; and

decline the transaction associated with the authorization request based on the machine learning model of the fraud monitoring system as trained based on the plurality of debiased embedding vectors.

12 . The computer program product of claim 11 , wherein the plurality of embedding vectors are vector representations of merchant identity embeddings generated from customer transaction data.

13 . The computer program product of claim 11 , wherein the bias score is calculated from a Word Embedding Association Test (WEAT).

14 . The computer program product of claim 11 , wherein the bias score is calculated from an Embedding Coherence Test (ECT).

15 . The computer program product of claim 11 , wherein the instructions, when executed by at least one processor, further cause the at least one processor to, before modifying the first mean vector and the second mean vector, determine an initial bias score by linearly projecting the embedding vectors of both the first cluster and the second cluster along an initial direction defined between the first mean vector and the second mean vector, and wherein:

the replacing of the first mean vector with the first candidate vector of the plurality of first candidate vectors is based on maximizing a difference between the initial bias score and the bias score of the first candidate vector; and

the replacing of the second mean vector with the second candidate vector of the plurality of second candidate vectors is based on maximizing a difference between the initial bias score and the bias score of the second candidate vector.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 7, 2023
From: DEV, SUNIPA; ZHENG, YAN; YEH, MICHAEL; WANG, JUNPENG; ZHANG, WEI; RATHORE, ARCHIT
To: VISA INTERNATIONAL SERVICE ASSOCIATION
Reel/Frame 064829/0372 →
Continuity (2)
Provisional Application 63167737 · Mar 30, 2021
Related Publication 20240160854A1 · May 16, 2024
References Cited (81)
US 10671942B1 · McGovern et al. · 2020 [cited by applicant]
US 11526508B1 · McCallie, Jr. · 2022 [cited by examiner]
US 20170372318A1 · Shami · 2017 [cited by examiner]
US 20190147371A1 · Deo · 2019 [cited by examiner]
US 20200314101A1 · Zhang et al. · 2020 [cited by applicant]
US 20210124780A1 · Silva · 2021 [cited by examiner]
US 20210150631A1 · Resheff · 2021 [cited by examiner]
US 20210350285A1 · Malur Srinivasan · 2021 [cited by examiner]
US 20220147713A1 · Garimella · 2022 [cited by examiner]
US 20220198470A1 · Melul · 2022 [cited by examiner]
US 20220245339A1 · Panda · 2022 [cited by examiner]
EP 3483797A1 · 2019 [cited by applicant]
Popovic et al. “Joint Multiclass Debiasing of Word Embeddings”. arXiv:2003.11520v1 [cs.CL] Mar. 9, 2020 (Year: 2020). [cited by examiner]
Celikkanat et al. “Tracking the Traces of Passivization and Negation in Contextualized Representations”. Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 136-148 O… [cited by examiner]
Ravfogel et al. “Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection”. arXiv:2004.07667v2 [cs. CL] Apr. 28, 2020 (Year: 2020). [cited by examiner]
Abadi et al., “TensorFlow: A System for Large-Scale Machine Learning”, Usenix, 2016, pp. 265-283. [cited by applicant]
Amershi et al., “ModelTracker: Redesigning Performance Analysis Tools for Machine Learning”, 2015, 10 pages. [cited by applicant]
Arrieta et al., “Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI”, 2019, pp. 1-72. [cited by applicant]
Bolukbasi et al., “Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings”, 30th Conference on Neural Information Processing Systems (NIPS 2016), 2016, pp. 1-9. [cited by applicant]
Cabrera et al., “FAIRVIS: Visual Analytics for Discovering Intersectional Bias in Machine Learning”, 2019, 11 pages. [cited by applicant]
Caliskan et al., “Semantics derived automatically from language corpora necessarily contain human biases”, 2017, 14 pages. [cited by applicant]
Chen et al., “OoDAnalyzer: Interactive Analysis of Out-of-Distribution Samples”, 2020, 16 pages. [cited by applicant]
Choo et al., “Visual Analytics for Explainable Deep Learning”, 10 pages, Aug. 2018. [cited by applicant]
Chuang et al., “Termite: Visualization Techniques for Assessing Textual Topic Models”, 2012, 4 pages. [cited by applicant]
Dev et al., “On Measuring and Mitigating Biased Inferences of Word Embeddings”, 2019, 11 pages. [cited by applicant]
Dev et al., “Closed Form Word Embedding Alignment”, 2020, 13 pages. [cited by applicant]
Dev et al., “Attenuating Bias in Word Vectors”, 2019, 17 pages. [cited by applicant]
Dev et al., “OSCAR: Orthogonal Subspace Correction and Rectification of Biases in Word Embeddings”, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, pp. 5034-5050. [cited by applicant]
Dev et al., “On Measuring and Mitigating Biased Inferences of Word Embeddings”, The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20), 2020, pp. 7659-7666. [cited by applicant]
Devlin et al., “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, Proceedings of NAACL-HLT, 2019, pp. 4171-4186. [cited by applicant]
Du et al., “Pcard: Personalized Restaurants Recommendation from Card Payment Transaction Records”, 2019, 7 pages. [cited by applicant]
Facets, “Facets—Visualization for ML datasets”, retrieved from https://paircode.github.io/facets/, Feb. 21, 2024. [cited by applicant]
Ghosh et al., “VisExPreS: A Visual Interactive Toolkit for User-Driven Evaluations of Embeddings”, IEEE Transactions on Visualization and Computer Graphics, 2022, pp. 2791-2807, vol. 28, No. 7. [cited by applicant]
Gonen et al., “Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them”, 2019, 6 pages. [cited by applicant]
Google, “Visually probe the behavior of trained machine learning models, with minimal coding”, What-If Tool, retrieved from https://pair-code.github.io/what-if-tool/, Feb. 22, 2024. [cited by applicant]
Heimerl et al., “embComp: Visual Interactive Comparison of Vector Embeddings”, 2021, pp. 1-16. [cited by applicant]
Heimerl et al., “Interactive Analysis of Word Vector Embeddings”, Eurographics Conference on Visualization (EuroVis), 2018, 13 pages, vol. 37, No. 3. [cited by applicant]
Hinterreiter et al., “ConfusionFlow: A model-agnostic visualization for temporal analysis of classifier confusion”, IEEE Transactions on Visualization and Computer Graphics, 2020, pp. 1-15. [cited by applicant]
Kahng et al., “Visual Exploration of Machine Learning Results using Data Cube Analysis”, 2016, 6 pages. [cited by applicant]
Krause et al., “INFUSE: Interactive Feature Selection for Predictive Modeling of High Dimensional Data”, IEEE Transactions on Visualization and Computer Graphics, 2014, pp. 1614-1623, vol. 20, No. 12. [cited by applicant]
Kusner et al., “Counterfactual Fairness”, 31st Conference on Neural Information Processing Systems (NIPS 2017), 2017, pp. 1-11. [cited by applicant]
Li et al., “EmbeddingVis: A visual Analytics Approach to Comparative Network Embedding Inspection”, 2018, 12 pages. [cited by applicant]
Liu et al., “Towards Better Analysis of Machine Learning Models: A Visual Analytics Perspective”, 2017, pp. 1-23. [cited by applicant]
Liu et al., “LatentVis: Investigating and Comparing Variational Auto-Encoders via Their Latent Space”, Proceedings of the CIKM 2020 Workshops, 2020, 10 pages. [cited by applicant]
Liu et al., “NLIZE: A Perturbation-Driven Visual Interrogation Tool for Analyzing and Interpreting Natural Language Inference Models”, 10 pages, Jan. 2019. [cited by applicant]
Liu et al., “Visual Exploration of Semantic Relationships in Neural Word Embeddings”, 10 pages. [cited by applicant]
Liu et al., “Latent Space Cartography: Visual Analysis of Vector Space Embeddings”, Eurographics Conference on Visualization (EuroVis), 2019, 12 pages, vol. 38, No. 3. [cited by applicant]
Liu et al., “Visual Interrogation of Attention-Based Models for Natural Language Inference and Machine Comprehension”, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (System Demon… [cited by applicant]
Liu et al., “Towards Better Analysis of Deep Convolutional Neural Networks”, IEEE Transactions on Visualization and Computer Graphics, 2017, pp. 91-100, vol. 23, No. 1. [cited by applicant]
Liu et al., “Visual Diagnosis of Tree Boosting Methods”, IEEE Transactions on Visualization and Computer Graphics, 2018, pp. 163-173, vol. 24, No. 1. [cited by applicant]
Liu et al., “Visualizing High-Dimensional Data: Advances in the Past Decade”, IEEE Transactions on Visualization and Computer Graphics, 2017, pp. 1249-1268, vol. 23, No. 3. [cited by applicant]
Liu et al., “StoryFlow: Tracking the Evolution of Stories”, IEEE Transactions on Visualization and Computer Graphics, 2013, pp. 2436-2445, vol. 19, No. 12. [cited by applicant]
Liu et al., “Online Visual Analytics of Text Streams”, IEEE Transactions on Visualization and Computer Graphics, 2016, pp. 2451-2466, vol. 22, No. 11. [cited by applicant]
May et al., “Guiding Feature Subset Selection with an Interactive Visualization”, IEEE Symposium on Visual Analytics Science and Technology, 2011, pp. 111-120. [cited by applicant]
Mcinnes et al., “UMAP: Uniform Manifold Approximation and Projection”, Journal of Open Source Software, 2018, pp. 1-2, vol. 3, No. 29. [cited by applicant]
Mcinnes et al., “UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction”, 2020, pp. 1-63. [cited by applicant]
Mikolov et al., “Distributed Representations of Words and Phrases and their Compositionality”, pp. 1-9. 2013. [cited by applicant]
Pedregosa et al., “Scikit-learn: Machine Learning in Python”, Journal of Machine Learning Research, 2011, pp. 2825-2830, vol. 12. [cited by applicant]
Pennington et al., “GloVe: Global Vectors for Word Representation”, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014, pp. 1532-1543. [cited by applicant]
Pezzotti et al., “DeepEyes: Progressive Visual Analytics for Designing Deep Neural Networks”, IEEE Transactions on Visualization and Computer Graphics, 2018, pp. 1-10. [cited by applicant]
Rathore et al., “TopoAct: Visually Exploring the Shape of Activations in Deep Learning”, 2021, pp. 1-18. [cited by applicant]
Rauber et al., “Visualizing the Hidden Activity of Artificial Neural Networks”, 2017, 10 pages. [cited by applicant]
Ravfogel et al., “Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection”, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 7237-7256. [cited by applicant]
Ren et al., “Squares: Supporting Interactive Performance Analysis for Multiclass Classifiers”, 10 pages, Jan. 2017. [cited by applicant]
Ribeiro et al., “Why Should I Trust You?' Explaining the Predictions of Any Classifier”, Proceedings of NAACL-HLT 2016 (Demonstrations), 2016, pp. 97-101. [cited by applicant]
Smilkov et al., “Embedding Projector: Interactive Visualization and Interpretation of Embeddings”, 30th Conference on Neural Information Processing Systems (NIPS), 2016, pp. 1-4. [cited by applicant]
Smith et al., “Hierarchie: Interactive Visualization for Hierarchical Topic Models”, Proceedings of the Workshop on Interactive Language Learning, Visualization, and Interfaces, 2014, pp. 71-78. [cited by applicant]
Smith et al., “Concurrent Visualization of Relationships between Words and Topics in Topic Models”, Proceedings of the Workshop on Interactive Language Learning, Visualization, and Interfaces, 2014, pp. 79-82. [cited by applicant]
Springenberg et al., “Striving for Simplicity: The All Convolutional Net”, 2015, pp. 1-14. [cited by applicant]
Van Der Maaten et al., “Visualizing Data using t-SNE”, Journal of Machine Learning Research, 2008, pp. 2579-2605, vol. 9. [cited by applicant]
Wang et al., “Feature Detection and Attenuation in Embeddings”, 2021, pp. 1-11. [cited by applicant]
Wang et al., “SentiView: Sentiment Analysis and Visualization for Internet Popular Topics”, IEEE Transactions on Human-Machine Systems, 2013, pp. 1-11. [cited by applicant]
Wang et al., “GANViz: A Visual Analytics Approach to Understand the Adversarial Game”, IEEE Transactions on Visualization and Computer Graphics, 2018, pp. 1-12. [cited by applicant]
Wang et al., “Double-Hard Debias: Tailoring Word Embeddings for Gender Bias Mitigation”, 2020, 11 pages. [cited by applicant]
Wang et al., “SCANViz: Interpreting the Symbol-Concept Association Captured by Deep Neural Networks through Visual Analytics”, 2020, 11 pages. [cited by applicant]
Wongsuphasawat et al., “Visualizing Dataflow Graphs of Deep Learning Models in TensorFlow”, 12 pages. Jan. 2018. [cited by applicant]
Yang et al., “Diagnosing Concept Drift with Visual Analytics”, 2020, 12 pages. [cited by applicant]
Yeh et al., “Merchant Category Identification Using Credit Card Transactions”, Visa Research, 2020, 9 pages. [cited by applicant]
Yeh et al., “Towards a Flexible Embedding Learning Framework”, Visa Research, 2020, 10 pages. [cited by applicant]
Yuan et al., “A survey of visual analytics techniques for machine learning”, Computational Visual Media, 2021, pp. 3-36, vol. 7, No. 1. [cited by applicant]
Zhao et al., “Learning Gender-Neutral Word Embeddings”, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 4847-4853. [cited by applicant]