IP Library Granted Patent US 11,321,638
Granted Patent B2
US 11,321,638 · App. 16/819,445 · Granted May 3, 2022

Interoperable smart AI enabled evaluation of models

Inventors: Mary Rudden (Denver, CO); Craig M. Trim (Ventura, CA); Shikhar Kwatra (Durham, NC); Stacey Heeley Ronaghan (Los Angeles, CA); William G. Dusch (Morrisville, NC)
Assignee: Kyndryl, Inc.
G06N20/00G06F9/451
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,321,638
App. No.
16/819,445
Granted
May 3, 2022
Kind
B2
Abstract

A machine learning model can be depicted as a graph. After identifying a familiarity level of a user, the graph of the machine learning model can be customized. The customization is based on the familiarity level of the user. The customized graph of the machine learning model is displayed to the user.

Claims (62)

1. A method, comprising:

creating a graphical overview of M machine learning models, wherein M is at least 2;

determining a familiarity level of a user, wherein the familiarity level of a user is a level of the user's technical knowledge of machine learning models;

customizing the graphical overview based on the familiarity level; an

displaying the graphical overview, wherein said displaying the graphical overview comprises displaying N nodes and multiple edges, wherein N is greater than 2, wherein each edge is a line connecting only two nodes of the N nodes, wherein the N nodes comprise M model nodes, one or more input nodes, and one or more hyperparameter nodes, wherein each model node represents a unique model of the M models, wherein each model node is connected to at least one input node by a unique edge of the multiple edges, wherein each model node is connected to at least one hyperparameter node by one unique edge of the multiple edges, wherein each input node represents one or more inputs to each model node connected to said each input node, and wherein each hyperparameter node represents one or more hyperparameters used by each model node connected to said each input node.

2. The method of claim 1 , further comprising:

detecting a user input;

determining, based on the user input, that a graph clarification is implicated; and

updating the graphical overview based on the graph clarification.

3. The method of claim 2 , further comprising updating, based on the input, the familiarity level of the user.

4. The method of claim 1 , further comprising:

determining that a storage device does not include a familiarity profile associated with the user; and

creating, responsive to the determining, a new familiarity profile associated with the user,

the creating including:

prompting a user to input familiarity information;

receiving a response to the prompting;

determining an initial familiarity level based on the response; and

storing the initial familiarity level in the new familiarity profile, wherein the determining the familiarity level of the user is based on the new familiarity profile.

5. The method of claim 1 , further comprising determining that a storage device includes a familiarity profile associated with the user, wherein the determining the familiarity level of the user is based on the familiarity profile.

6. A system comprising: a memory; and a central processing unit (CPU) including one or more CPU cores configured to:

create a graphical overview of M machine learning models, wherein M is at least 2;

determine a familiarity level of a user, wherein the familiarity level of a user is a level of the user's technical knowledge of machine learning models;

customize the graphical overview based on the familiarity level; and

display the graphical overview, wherein said displaying the graphical overview comprises displaying N nodes and multiple edges, wherein N is greater than 2, wherein each edge is a line connecting only two nodes of the N nodes, wherein the N nodes comprise M model nodes, one or more input nodes, and one or more hyperparameter nodes, wherein each model node represents a unique model of the M models, wherein each model node is connected to at least one input node by a unique edge of the multiple edges, wherein each model node is connected to at least one hyperparameter node by one unique edge of the multiple edges, wherein each input node represents one or more inputs to each model node connected to said each input node, and wherein each hyperparameter node represents one or more hyperparameters used by each model node connected to said each input node.

7. The system of claim 6 , wherein the CPU is further configured to:

detect a user input;

determine, based on the user input, that a graph clarification is implicated; and

update the graphical overview based on the graph clarification.

8. The system of claim 7 , wherein the CPU is further configured to update, based on the input, the familiarity level of the user.

9. The system of claim 6 , wherein the CPU is further configured to:

determine that a storage device does not include a familiarity profile associated with the user; and

create, responsive to the determining, a new familiarity profile associated with the user,

the creating including:

prompting a user to input familiarity information;

receiving a response to the prompting;

determining an initial familiarity level based on the response; and

storing the initial familiarity level in the new familiarity profile, wherein the determining the familiarity level of the user is based on the new familiarity profile.

10. The system of claim 6 , wherein the CPU is further configured to determine that a storage device includes a familiarity profile associated with the user, wherein the determining the familiarity level of the user is based on the familiarity profile.

11. A computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:

create a graphical overview of M machine learning models, wherein M is at least 2;

determine a familiarity level of a user, wherein the familiarity level of a user is a level of the user's technical knowledge of machine learning models;

customize the graphical overview based on the familiarity level; and

display the graphical overview, wherein said displaying the graphical overview comprises displaying N nodes and multiple edges, wherein N is greater than 2, wherein each edge is a line connecting only two nodes of the N nodes, wherein the N nodes comprise M model nodes, one or more input nodes, and one or more hyperparameter nodes, wherein each model node represents a unique model of the M models, wherein each model node is connected to at least one input node by a unique edge of the multiple edges, wherein each model node is connected to at least one hyperparameter node by one unique edge of the multiple edges, wherein each input node represents one or more inputs to each model node connected to said each input node, and wherein each hyperparameter node represents one or more hyperparameters used by each model node connected to said each input node.

12. The computer program product of claim 11 , wherein the instructions further cause the computer to:

detect a user input;

determine, based on the user input, that a graph clarification is implicated; and

update the graphical overview based on the graph clarification.

13. The computer program product of claim 12 , wherein the instructions further cause the computer to update, based on the input, the familiarity level of the user.

14. The computer program product of claim 11 , wherein the instructions further cause the computer to:

determine that a storage device does not include a familiarity profile associated with the user; and

create, responsive to the determining, a new familiarity profile associated with the user,

the creating including:

prompting a user to input familiarity information;

receiving a response to the prompting;

determining an initial familiarity level based on the response; and

storing the initial familiarity level in the new familiarity profile, wherein the determining the familiarity level of the user is based on the new familiarity profile.

15. The method of claim 1 , wherein said displaying the graphical overview further comprises displaying N information boxes, and wherein each information box includes information about a respective node of the N nodes and is not physically connected to the respective node.

16. The system of claim 6 , wherein said display of the graphical overview further comprises display of N information boxes, and wherein each information box includes information about a respective node of the N nodes and is not physically connected to the respective node.

17. The computer program product of claim 11 , wherein said display of the graphical overview further comprises display of N information boxes, and wherein each information box includes information about a respective node of the N nodes and is not physically connected to the respective node.

18. The method of claim 1 , wherein the N nodes further comprise M score nodes, wherein each score node is connected to a unique model node of the M model nodes by one of the edges, and wherein each score node specifies a numerical score for the unique model node and a metric that the numerical score represents.

19. The system of claim 6 , wherein the N nodes further comprise M score nodes, wherein each score node is connected to a unique model node of the M model nodes by one of the edges, and wherein each score node specifies a numerical score for the unique model node and a metric that the numerical score represents.

20. The computer program product of claim 11 , wherein the N nodes further comprise M score nodes, wherein each score node is connected to a unique model node of the M model nodes by one of the edges, and wherein each score node specifies a numerical score for the unique model node and a metric that the numerical score represents.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 18, 2021
From: INTERNATIONAL BUSINESS MACHINES CORPORATION
To: KYNDRYL, INC.
Reel/Frame 058213/0912 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 16, 2020
From: RUDDEN, MARY; TRIM, CRAIG M.; KWATRA, SHIKHAR; RONAGHAN, STACEY HEELEY; DUSCH, WILLIAM G.
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 052122/0757 →
Continuity (1)
Related Publication 20210287133A1 · Sep 16, 2021