IP Library › Granted Patent US 12688687
Granted Patent B2
US 12688687 · App. 18/438,486 · Granted Jul 21, 2026

Systems and methods for explaining machine learning models

Inventors: Ziyan Wu (Lexington, MA); Meng Zheng (Cambridge, MA); Benjamin Planche (Briarwood, NY); Zhongpai Gao (Rowley, MA); Abhishek Sharma (Boston, MA); Terrence Chen (Lexington, MA)
Assignee: Shanghai United Imaging Intelligence Co., Ltd.
G06V10/86G06V10/764G06V10/7715G06V10/774G06V10/82
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12688687
App. No.
18/438,486
Granted
Jul 21, 2026
Kind
B2
Abstract

The decision process of a first machine learning (ML) model may be explained based on a second ML model implemented on an apparatus. The apparatus may obtain a prediction about an image made based on the first ML model. The apparatus may further determine visual concepts associated with the image that may have been used by the first ML model to make the prediction, and determine respective contributions of the visual concepts to the prediction made by the first ML model. The apparatus may then generate, based on the second ML model, a textual description that explains the respective contributions of the visual concepts to the prediction made by the first ML model. The second ML model may determine respective image features associated with the visual concepts, map the determined image features to corresponding text features, and generate the textual description based at least on the text features.

Claims (35)

1 . An apparatus, comprising:

one or more processors configured to:

obtain a prediction about an image, wherein the prediction is made based on a first machine learning (ML) model;

determine visual concepts associated with the image that are used by the first ML model to make the prediction;

determine respective contributions of the visual concepts to the prediction made by the first ML model; and

generate, based on a second ML model, a textual description that explains the respective contributions of the visual concepts to the prediction made by the first ML model;

wherein the second ML model includes a graph reasoning neural network pre-trained to generate a graph representation that indicates a reasoning process of the first ML model when making the prediction about the image, the graph representation comprising nodes and edges, each node representing a respective one of the visual concepts, each edge connecting two nodes and representing a dependency of the respective visual concepts represented by the two nodes during the reasoning process of the first ML model, and wherein the second ML model further includes a vision-language model pre-trained to determine respective image features associated with the visual concepts, map the determined image features to corresponding text features, and generate the textual description based at least on the text features.

2 . The apparatus of claim 1 , wherein the vision-language model is trained using a plurality of image-text pairs from which the vision-language model learns a mapping between the image features associated with the visual concepts and the text features used to generate the textual description.

3 . The apparatus of claim 2 , wherein the vision-language model learns, from the plurality of image-text pairs, a joint embedding space in which the image features-associated with the visual concepts are mapped to the text features used to generate the textual description.

4 . The apparatus of claim 1 , wherein each of the visual concepts corresponds to one or more patches of the image.

5 . The apparatus of claim 1 , wherein the vision-language model is implemented using at least an encoder and a decoder, wherein the encoder is configured to determine the respective image features associated with the visual concepts, and wherein the decoder is configured to generate the textual description that explains the respective contributions of the visual concepts to the prediction made by the first ML model.

6 . The apparatus of claim 1 , wherein the graph reasoning neural network is trained using a knowledge distillation technique via which the graph reasoning neural network learns the reasoning process of the first ML model.

7 . The apparatus of claim 6 , wherein, during the training of the graph reasoning neural network, parameters of the graph reasoning neural network are optimized by forcing the graph reasoning neural network to make a same prediction about an image as the first ML model when the graph reasoning neural network is given a training graph representation comprising nodes representing visual concepts extracted from the image and edges representing dependencies of the visual concepts.

8 . The apparatus of claim 1 , wherein the one or more processors being configured to determine the respective contributions of the visual concepts to the prediction made by the first ML model comprises the one or more processors being configured to:

determine, based on the graph representation generated by the graph reasoning neural network, respective contribution scores of the visual concepts to the prediction; and

add an indication of the contribution scores to the graph representation.

9 . The apparatus of claim 1 , wherein the vision-language model is configured to generate the textual description that explains the respective contributions of the visual concepts to the prediction made by the first ML model further based on the graph representation generated by the graph reasoning neural network.

10 . The apparatus of claim 1 , wherein the prediction about the image is a classification of the image.

11 . A method for explaining a decision process of a machine learning (ML) model, the method comprising:

obtaining a prediction about an image, wherein the prediction is made based on a first ML model;

determining visual concepts associated with the image that are used by the first ML model to make the prediction;

determining respective contributions of the visual concepts to the prediction made by the first ML model; and

generating, based on a second ML model, a textual description that explains the respective contributions of the visual concepts to the prediction made by the first ML model;

wherein the second ML model includes a graph reasoning neural network pre-trained to generate a graph representation that indicates a reasoning process of the first ML model when making the prediction about the image, the graph representation comprising nodes and edges, each node representing a respective one of the visual concepts, each edge connecting two nodes and representing a dependency of the respective visual concepts represented by the two nodes during the reasoning process of the first ML model, and wherein the second ML model further includes a vision-language model pre-trained to determine respective image features associated with the visual concepts, map the determined image features to corresponding text features, and generate the textual description based at least on the text features.

12 . The method of claim 11 , wherein the vision-language model is trained using a plurality of image-text pairs from which the vision-language model learns a mapping between the image features associated with the visual concepts and the text features used to generate the textual description.

13 . The method of claim 12 , wherein the vision-language model learns, from the plurality of image-text pairs, a joint embedding space in which the image features associated with the visual concepts are mapped to the text features used to generate the textual description.

14 . The method of claim 11 , wherein each of the visual concepts corresponds to one or more patches of the image.

15 . The method of claim 11 , wherein the vision-language model is implemented using at least an encoder and a decoder, wherein the encoder is configured to determine the respective image features associated with the visual concepts, and wherein the decoder is configured to generate the textual description that explains the respective contributions of the visual concepts to the prediction made by the first ML model.

16 . The method of claim 11 , wherein the graph reasoning neural network is trained using a knowledge distillation technique via which the graph reasoning neural network learns the reasoning process of the first ML model.

17 . The method of claim 16 , wherein, during the training of the graph reasoning neural network, parameters of the graph reasoning neural network are optimized by forcing the graph reasoning neural network to make a same prediction about an image as the first ML model when the graph reasoning neural network is given a training graph representation comprising nodes representing visual concepts extracted from the image and edges representing dependencies of the visual concepts.

18 . The method of claim 11 , wherein determining the respective contributions of the visual concepts to the prediction made by the first ML model comprises:

determining, based on the graph representation generated by the graph reasoning neural network, respective contribution scores of the visual concepts to the prediction; and

adding an indication of the contribution scores to the graph representation.

19 . The method of claim 11 , wherein the vision-language model is configured to generate the textual description that explains the respective contributions of the visual concepts to the prediction made by the first ML model further based on the graph representation generated by the graph reasoning neural network.

20 . A non-transitory computer-readable medium comprising instructions that, when executed by a processor included in a computing device, cause the processor to implement the method of claim 11 .