IP Library Granted Patent US 11,544,604
Granted Patent B2
US 11,544,604 · App. 16/597,336 · Granted Jan 3, 2023

Adaptive model insights visualization engine for complex machine learning models

Inventors: Ashok Pancily Poothiyot (San Jose, CA); Vidit Bhatia (San Jose, CA); Matthew Colon (San Jose, CA)
Assignee: Adobe Inc.
G06N5/045G06F8/20G06N20/00G06F8/30G06F8/34
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,544,604
App. No.
16/597,336
Granted
Jan 3, 2023
Kind
B2
Abstract

The present disclosure relates to systems, methods, and non-transitory computer-readable media for utilizing a parameterized notebook to adaptively generate visualizations regarding machine-learning models. In particular, the disclosed systems can generate a parameterized notebook based on a user-defined visualization recipe and provide parameter values that correspond to the machine-learning model to the parameterized notebook. Upon execution of the user-defined visualization recipe via the parameterized notebook, the disclosed systems can extract visualization data corresponding to the machine-learning model from the parameterized notebook. In addition, the disclosed systems can generate visualizations based on the visualization data and provide the generated visualizations for display in a graphical user interface.

Claims (68)

1. A computer-implemented method comprising:

generating, based on a user-defined visualization recipe, a headless parameterized notebook, wherein the user-defined visualization recipe comprises computer executable instructions in a first runtime language;

providing, to the headless parameterized notebook for executing the user-defined visualization recipe, a first set of parameter values from executing a machine-learning model comprising instructions in a second runtime language different from the first runtime language;

upon execution of the user-defined visualization recipe via the headless parameterized notebook, extracting visualization data corresponding to the machine-learning model from the headless parameterized notebook; and

generating, for display in a graphical user interface, a visualization based on the visualization data.

2. The computer-implemented method of claim 1 , further comprising:

detecting, by tracking execution runs, an interaction history with one or more machine-learning models; and

training a suggestion machine-learning model to generate suggestions based on the interaction history.

3. The computer-implemented method of claim 2 , further comprising:

applying the trained suggestion machine-learning model to at least one of the machine-learning model, the first set of parameter values, or the user-defined visualization recipe to generate a suggestion comprising at least one of a suggested visualization, a suggested visualization recipe, or a modification to the machine-learning model.

4. The computer-implemented method of claim 1 , wherein the visualization comprises at least one of a portable network graphics output or a hypertext markup language output.

5. A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause a computer system to perform operations comprising:

generating, based on a user-defined visualization recipe, a headless parameterized notebook, wherein the user-defined visualization recipe comprises computer-executable instructions in a first runtime language;

providing, to the headless parameterized notebook for executing the user-defined visualization recipe, a set of parameter values from executing a machine-learning model comprising instructions in a second runtime language different from the first runtime language;

upon execution of the user-defined visualization recipe via the headless parameterized notebook, extracting visualization data corresponding to the machine-learning model from the headless parameterized notebook; and

generating, for display in a graphical user interface, a visualization based on the visualization data.

6. The non-transitory computer-readable medium of claim 5 , wherein providing the set of parameter values from executing the machine-learning model is based on identifying one or more of new parameter values, completion of an execution run, lapsing of a time period, or a request from a client device.

7. The non-transitory computer-readable medium of claim 5 , further comprising instructions that, when executed by the at least one processor, cause the computer system to perform operations comprising:

re-executing the machine-learning model;

generating a second set of parameter values corresponding to re-executing the machine-learning model; and

providing the second set of parameter values to the headless parameterized notebook for re-executing the user-defined visualization recipe.

8. The non-transitory computer-readable medium of claim 7 , further comprising instructions that, when executed by the at least one processor, cause the computer system to perform operations comprising:

upon re-execution of the user-defined visualization recipe via the headless parameterized notebook, extracting a second set of visualization data corresponding to re-executing the machine-learning model from the headless parameterized notebook; and

generating, for display in the graphical user interface, a new visualization based on the second set of visualization data.

9. The non-transitory computer-readable medium of claim 5 , further comprising instructions that, when executed by the at least one processor, cause the computer system to perform operations comprising:

generating the visualization by converting the visualization data in a first format to a second format,

wherein the first format comprises at least one of a tabular format, a JavaScript format, a JavaScript object notation format, or a scalable vector graphics format; and

wherein the second format comprises at least one of a portable network graphics format or a hypertext markup language format.

10. The non-transitory computer-readable medium of claim 5 , further comprising instructions that, when executed by the at least one processor, cause the computer system to perform operations comprising:

detecting, by tracking execution runs, an interaction history with one or more machine-learning models; and

training a suggestion machine-learning model to generate suggestions based on the interaction hi story.

11. The non-transitory computer-readable medium of claim 10 , further comprising instructions that, when executed by the at least one processor, cause the computer system to perform operations comprising:

applying the trained suggestion machine-learning model to at least one of the machine-learning model, the set of parameter values, or the user-defined visualization recipe to generate a suggestion comprising at least one of:

a suggested visualization;

a suggested visualization recipe; or

a modification to the machine-learning model.

12. The non-transitory computer-readable medium of claim 11 , further comprising instructions that, when executed by the at least one processor, cause the computer system to perform operations comprising automatically implementing the suggestion from the suggestion machine-learning model.

13. A system comprising:

one or more memory devices comprising a machine-learning model and a user-defined visualization recipe; and

one or more server devices that cause the system to:

execute the machine-learning model to determine a set of parameter values corresponding to the machine-learning model, wherein the machine-learning model comprises computer-executable instructions in a first runtime language;

generate, based on the user-defined visualization recipe, a headless parameterized notebook, wherein the user-defined visualization recipe comprises computer-executable instructions in a second runtime language different from the first runtime language;

provide the set of parameter values to the headless parameterized notebook for applying the user-defined visualization recipe to the set of parameter values;

upon execution of the user-defined visualization recipe via the headless parameterized notebook, extract visualization data corresponding to the machine-learning model from the headless parameterized notebook; and

generate, for display in a graphical user interface, a visualization based on the visualization data.

14. The system of claim 13 , wherein the one or more server devices further cause the system to:

execute a second machine-learning model comprising computer-executable instructions in a third runtime language different from the second runtime language;

generate a second set of parameter values corresponding to executing the second machine-learning model; and

provide the second set of parameter values to the headless parameterized notebook for executing the user-defined visualization recipe in the second runtime language.

15. The system of claim 13 , wherein the one or more server devices further cause the system to:

re-execute the machine-learning model;

generate a second set of parameter values corresponding to re-executing the machine-learning model; and

provide the second set of parameter values to the headless parameterized notebook for re-executing the user-defined visualization recipe.

16. The system of claim 15 , wherein the one or more server devices further cause the system to:

upon re-execution of the user-defined visualization recipe via the headless parameterized notebook, extract a second set of visualization data corresponding to re-executing the machine-learning model from the headless parameterized notebook; and

generate, for display in the graphical user interface, a new visualization based on the second set of visualization data.

17. The system of claim 13 , wherein the one or more server devices further cause the system to:

generate the visualization by converting the visualization data in a first format to a second format,

wherein the first format comprises at least one of a JavaScript format, a JavaScript object notation format, or a scalable vector graphics format; and

wherein the second format comprises at least one of a portable network graphics format or a hypertext markup language format.

18. The system of claim 13 , wherein the one or more server devices further cause the system to utilize a suggestion machine-learning model to:

detect, by tracking execution runs, an interaction history with one or more machine-learning models; and

train a suggestion machine-learning model to generate suggestions based on the interaction history.

19. The system of claim 18 , wherein the one or more server devices further cause the system to apply the trained suggestion machine-learning model to at least one of the machine-learning model, the set of parameter values, or the user-defined visualization recipe to generate a suggestion comprising at least one of:

a suggested visualization;

a suggested visualization recipe; or

a modification to the machine-learning model.

20. The system of claim 19 , wherein the one or more server devices further cause the system to automatically implement the suggestion from the suggestion machine-learning model.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 9, 2019
From: POOTHIYOT, ASHOK PANCILY; BHATIA, VIDIT; COLON, MATTHEW
To: ADOBE INC.
Reel/Frame 050667/0876 →
Continuity (1)
Related Publication 20210110288A1 · Apr 15, 2021
Cited By (3)
US 12,386,667 US 12,481,534 US 12,657,058