IP Library › Granted Patent US 12664471
Granted Patent B2
US 12664471 · App. 18/087,456 · Granted Jun 23, 2026

Machine learning model analysis

Inventors: Narayanan Unny Edakunni (Bengaluru, IN); Lalitha K S (Bengaluru, IN); Utkarsh Tekriwal (Bengaluru, IN)
Assignee: American Express Travel Related Services Company, Inc.
G06N20/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12664471
App. No.
18/087,456
Granted
Jun 23, 2026
Kind
B2
Abstract

Disclosed are various embodiments for analyzing machine learning models. A selection is obtained of a first tuple comprising a first feature vector and a first result generated by a machine learning model and a second tuple comprising a second feature vector and a second result generated by the machine learning model. Then, a plurality of emulated feature vectors are generated. Next, a plurality of emulated results are generated. Subsequently, a plurality of emulated decision instances are generated. Next, a decision tree is built based at least in part on the first tuple, the second tuple, and the plurality of emulated decision instances. Finally, an importance of each feature on the decision tree is computed.

Claims (41)

1 . A system, comprising:

a computing device comprising a processor and a memory; and

machine-readable instructions stored in the memory that, when executed by the processor, cause the computing device to at least:

obtain a selection of a first tuple comprising a first feature vector and a first result generated by a machine learning model and a second tuple comprising a second feature vector and a second result generated by the machine learning model;

generate a plurality of emulated feature vectors, each of the plurality of emulated feature vectors comprising a unique combination of features from the first feature vector and the second feature vector;

generate a plurality of emulated results, each of the plurality of emulated results resulting from providing a respective one of the plurality of emulated feature vectors to the machine learning model;

generate a plurality of emulated decision instances, each of the plurality of emulated decision instances comprising one of the additional predictions and the respective one of the plurality of emulated feature vectors;

build a decision tree based at least in part on the first tuple, the second tuple, and the plurality of emulated decision instances; and

perform a Tree SHAP (Shapley Additive explanation) analysis on the decision tree to compute an importance of each feature on the decision tree.

2 . The system of claim 1 , wherein the machine-readable instructions further cause the computing device to at least:

rank each feature based at least in part on the importance of the feature on the decision tree; and

generate a report that identifies each feature in ranked order and the importance of the feature on the decision tree.

3 . The system of claim 2 , wherein the machine-readable instructions further cause the computing device to at least provide the report to client device in data communication with the computing device.

4 . The system of claim 1 , wherein the machine-readable instructions that cause the computing device to generate the plurality of emulated feature vectors further cause the computing device to randomly generate a subset of feature vectors from the set of possible feature vectors.

5 . The system of claim 1 , wherein the machine-readable instructions further cause the computing device to at least obtain a selection of features of the machine learning model to include in the first feature vector, the second feature vector and the plurality of emulated feature vectors.

6 . The system of claim 1 , wherein the decision tree is a regression tree.

7 . A method, comprising:

obtaining a selection of a first tuple comprising a first feature vector and a first result generated by a machine learning model and a second tuple comprising a second feature vector and a second result generated by the machine learning model;

generating a plurality of emulated feature vectors, each of the plurality of emulated feature vectors comprising a unique combination of features from the first feature vector and the second feature vector;

generating a plurality of emulated results, each of the plurality of emulated results resulting from providing a respective one of the plurality of emulated feature vectors to the machine learning model;

generating a plurality of emulated decision instances, each of the plurality of emulated decision instances comprising one of the additional predictions and the respective one of the plurality of emulated feature vectors; building a decision tree based at least in part on the first tuple, the second tuple, and the plurality of emulated decision instances; and

performing a Tree SHAP (Shapley Additive explanation) analysis on the decision tree to compute an importance of each feature on the decision tree.

8 . The method of claim 7 , further comprising:

ranking each feature based at least in part on the importance of the feature on the decision tree; and

generating a report that identifies each feature in ranked order and the importance of the feature on the decision tree.

9 . The method of claim 8 , further comprising providing the report to client device in data communication with a computing device.

10 . The method of claim 7 , wherein generating the plurality of emulated feature vectors further comprises randomly generating a subset of feature vectors from the set of possible feature vectors.

11 . The method of claim 7 , further comprising obtaining a selection of features of the machine learning model to include in the first feature vector, the second feature vector and the plurality of emulated feature vectors.

12 . The method of claim 7 , wherein the decision tree is a regression tree.

13 . A non-transitory, computer-readable medium, comprising machine-readable instructions that, when executed by a processor of a computing device, cause the computing device to at least:

obtain a selection of a first tuple comprising a first feature vector and a first result generated by a machine learning model and a second tuple comprising a second feature vector and a second result generated by the machine learning model;

generate a plurality of emulated feature vectors, each of the plurality of emulated feature vectors comprising a unique combination of features from the first feature vector and the second feature vector;

generate a plurality of emulated results, each of the plurality of emulated results resulting from providing a respective one of the plurality of emulated feature vectors to the machine learning model;

generate a plurality of emulated decision instances, each of the plurality of emulated decision instances comprising one of the additional predictions and the respective one of the plurality of emulated feature vectors; build a decision tree based at least in part on the first tuple, the second tuple, and the plurality of emulated decision instances; and

perform a Tree SHAP (Shapley Additive explanation) analysis on the decision tree to compute an importance of each feature on the decision tree.

14 . The non-transitory, computer-readable medium of claim 13 , wherein the machine-readable instructions further cause the computing device to at least:

rank each feature based at least in part on the importance of the feature on the decision tree; and

generate a report that identifies each feature in ranked order and the importance of the feature on the decision tree.

15 . The non-transitory, computer-readable medium of claim 13 , wherein the machine-readable instructions that cause the computing device to generate the plurality of emulated feature vectors further cause the computing device to randomly generate a subset of feature vectors from the set of possible feature vectors.

16 . The non-transitory, computer-readable medium of claim 13 , wherein the machine-readable instructions further cause the computing device to at least obtain a selection of features of the machine learning model to include in the first feature vector, the second feature vector and the plurality of emulated feature vectors.

17 . The non-transitory, computer-readable medium of claim 13 , wherein the decision tree is a regression tree.