IP Library Granted Patent US 11,995,573
Granted Patent B2
US 11,995,573 · App. 18/305,278 · Granted May 28, 2024

Artificial intelligence system providing interactive model interpretation and enhancement tools

Inventors: Shikhar Gupta (Seattle, WA); Shriram Venkataramana (Seattle, WA); Sri Kaushik Pavani (Redmond, WA); Sunny Dasgupta (Redmond, WA)
Assignee: Amazon Technologies, Inc
G06N5/04G06F16/285G06N20/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,995,573
App. No.
18/305,278
Granted
May 28, 2024
Kind
B2
Abstract

An interactive interpretation session with respect to a first version of a machine learning model is initiated. In the session, indications of factors contributing to a prediction decision are provided, as well indications of candidate model enhancement actions. In response to received input, an enhancement action is implemented to obtain a second version of the model. The second version of the model is stored.

Claims (47)

1. A computer-implemented method, comprising:

causing to be presented, via one or more programmatic interfaces by a service of a cloud computing environment, a prediction generated by a first machine learning model for a particular input record;

obtaining, by the service via the one or more programmatic interfaces, a request for additional information pertaining to the prediction generated for the particular input record; and

causing to be presented, by the service via the one or more programmatic interfaces, in response to said obtaining, an indication of (a) one or more other input records of the first machine learning model which meet a similarity criterion with respect to the particular input record and (b) respective predictions generated by the first machine learning model with respect to individual ones of the one or more other input records.

2. The computer-implemented method as recited in claim 1 , wherein the first machine learning model comprises a classification model.

3. The computer-implemented method as recited in claim 1 , further comprising:

causing to be presented, by the service via the one or more programmatic interfaces, an indication of an importance, with respect to generation of the prediction by the first machine learning model, of a particular feature of the particular input record.

4. The computer-implemented method as recited in claim 1 , wherein the prediction indicates a particular class, selected from a plurality of classes, to which the particular input record is predicted to belong, the computer-implemented method further comprising:

causing to be presented, by the service via the one or more programmatic interfaces, an indication of an importance of a particular feature of individual ones of a plurality of input records with respect to classification of the plurality of input records into the particular class by the first machine learning model.

5. The computer-implemented method as recited in claim 1 , further comprising:

causing to be presented, by the service via the one or more programmatic interfaces, a representation of a confusion matrix of the first machine learning model.

6. The computer-implemented method as recited in claim 1 , wherein the first machine learning model comprises a modified version of a second machine learning model, the computer-implemented method further comprising:

causing to be presented, by the service via the one or more programmatic interfaces, (a) a representation of one or more changes made which were made to the second machine learning model to obtain the first machine learning model and (b) a representation of a difference in a quality metric between the second machine learning model and the first machine learning model.

7. The computer-implemented method as recited in claim 1 , further comprising:

causing to be presented, by the service via the one or more programmatic interfaces, an indication of a candidate model enhancement action applicable to the first machine learning model; and

generating, by the service, an enhanced version of the first machine learning model, based at least in part on implementing, in response to input received via the one or more programmatic interfaces, the candidate model enhancement action with respect to the first machine learning model.

8. A system, comprising:

one or more computing devices;

wherein the one or more computing devices include instructions that upon execution on or across the one or more computing devices:

cause to be presented, via one or more programmatic interfaces by a service of a cloud computing environment, a prediction generated by a first machine learning model for a particular input record;

obtain, by the service via the one or more programmatic interfaces, a request for additional information pertaining to the prediction generated for the particular input record; and

cause to be presented, by the service via the one or more programmatic interfaces, in response to the request, an indication of (a) one or more other input records of the first machine learning model which meet a similarity criterion with respect to the particular input record and (b) respective predictions generated by the first machine learning model with respect to individual ones of the one or more other input records.

9. The system as recited in claim 8 , wherein the first machine learning model comprises a classification model.

10. The system as recited in claim 8 , wherein the one or more computing devices include further instructions that upon execution on or across the one or more computing devices:

cause to be presented, by the service via the one or more programmatic interfaces, an indication of an importance, with respect to generation of the prediction by the first machine learning model, of a particular feature of the particular input record.

11. The system as recited in claim 8 , wherein the prediction indicates a particular class, selected from a plurality of classes, to which the particular input record is predicted to belong, and wherein the one or more computing devices include further instructions that upon execution on or across the one or more computing devices:

cause to be presented, by the service via the one or more programmatic interfaces, an indication of an importance of a particular feature of individual ones of a plurality of input records with respect to classification of the plurality of input records into the particular class by the first machine learning model.

12. The system as recited in claim 8 , wherein the one or more computing devices include further instructions that upon execution on or across the one or more computing devices:

cause to be presented, by the service via the one or more programmatic interfaces, a representation of a confusion matrix of the first machine learning model.

13. The system as recited in claim 8 , wherein the first machine learning model comprises a modified version of a second machine learning model, and wherein the one or more computing devices include further instructions that upon execution on or across the one or more computing devices:

cause to be presented, by the service via the one or more programmatic interfaces, (a) a representation of one or more changes made which were made to the second machine learning model to obtain the first machine learning model and (b) a representation of a difference in a quality metric between the second machine learning model and the first machine learning model.

14. The system as recited in claim 8 , wherein the one or more computing devices include further instructions that upon execution on or across the one or more computing devices:

cause to be presented, by the service via the one or more programmatic interfaces, an indication of a candidate model enhancement action applicable to the first machine learning model; and

generate, by the service, an enhanced version of the first machine learning model, based at least in part on implementing, in response to input received via the one or more programmatic interfaces, the candidate model enhancement action with respect to the first machine learning model.

15. One or more non-transitory computer-accessible storage media storing program instructions that when executed on or across one or more processors:

cause to be presented, via one or more programmatic interfaces by a service of a cloud computing environment, a prediction generated by a first machine learning model for a particular input record;

obtain, by the service via the one or more programmatic interfaces, a request for additional information pertaining to the prediction generated for the particular input record; and

cause to be presented, by the service via the one or more programmatic interfaces, in response to the request, an indication of (a) one or more other input records of the first machine learning model which meet a similarity criterion with respect to the particular input record and (b) respective predictions generated by the first machine learning model with respect to individual ones of the one or more other input records.

16. The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein the first machine learning model comprises a classification model.

17. The one or more non-transitory computer-accessible storage media as recited in claim 15 , storing further program instructions that when executed on or across the one or more processors:

cause to be presented, by the service via the one or more programmatic interfaces, an indication of an importance, with respect to generation of the prediction by the first machine learning model, of a particular feature of the particular input record.

18. The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein the prediction indicates a particular class, selected from a plurality of classes, to which the particular input record is predicted to belong, and wherein the one or more non-transitory computer-accessible storage media store further program instructions that when executed on or across the one or more processors:

cause to be presented, by the service via the one or more programmatic interfaces, an indication of an importance of a particular feature of individual ones of a plurality of input records with respect to classification of the plurality of input records into the particular class by the first machine learning model.

19. The one or more non-transitory computer-accessible storage media as recited in claim 15 , storing further program instructions that when executed on or across the one or more processors:

cause to be presented, by the service via the one or more programmatic interfaces, a representation of a confusion matrix of the first machine learning model.

20. The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein the first machine learning model comprises a modified version of a second machine learning model, and wherein the one or more non-transitory computer-accessible storage media store further program instructions that when executed on or across the one or more processors:

cause to be presented, by the service via the one or more programmatic interfaces, (a) a representation of one or more changes made which were made to the second machine learning model to obtain the first machine learning model and (b) a representation of a difference in a quality metric between the second machine learning model and the first machine learning model.

Continuity (2)
Continuation 16742746 · Jan 14, 2020
Related Publication 20230252325A1 · Aug 10, 2023
Cited By (1)
US 12,475,390